Simulation system, processing method, and information storage medium for changing a display object in response to a movement of a field of view

ABSTRACT

A simulation system includes a processor including hardware. The processor performs: a virtual space setting process of setting a virtual space in which a plurality of objects are arranged; a virtual camera control process of controlling a virtual camera corresponding to a point-of-view of a user wearing a head mounted display; and a display process of generating an image as viewed from the virtual camera in the virtual space as a display image on the head mounted display. In the virtual space setting process, the processor arranges at least one information display object in a line-of-sight direction of the user, and in the display process, when it is determined that a given change condition is satisfied by a change in the point-of-view of the user, the processor performs a change process of a display mode of the information display object.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2017/032827, having an international filing date of Sep. 12,2017, which designated the United States, the entirety of which isincorporated herein by reference. Japanese Patent Application No.2016-179890 filed on Sep. 14, 2016 is also incorporated herein byreference in its entirety.

BACKGROUND

The disclosure relates to a simulation system, a processing method, aninformation storage medium, and the like.

Systems using head mounted displays (HMDs) have conventionally beenknown. The system enables a user wearing the HMD on his or her head toexperience a virtual reality (VR) world by watching an image displayedon a screen of the HMD. For example, Japanese Patent ApplicationPublication No. 1995-261112 and Japanese Patent Application PublicationNo. 2012-123812 disclose conventional techniques for such a system usingsuch an HMD.

Japanese Patent Application Publication No. 1995-261112 discloses amethod by which, when the user's head enters into a predeterminedfluctuation state, the liquid crystal shutter of the HMD is opened toswitch visual information provided to the user to surroundingenvironmental visual information. Japanese Patent ApplicationPublication No. 2012-123812 discloses a method by which the motion ofthe user's head is identified by a motion detection unit, the processcorresponding to the angular speed is executed as the user desires, andthe return motion of the user's head is not reflected on the process sothat the operation by the motion of the user's head can be performed inan accurate manner.

In the system using an HMD, an image as viewed from a virtual camera ina virtual space is displayed on the HMD. With such an image displayed onthe HMD, a vast VR space spreads over the entire field of view of auser, whereby virtual reality for the user can be largely improved.

On the other hand, in the system using an HMD, it is desired to displayvarious kinds of information such as descriptive information and statusinformation to the user.

If such information is not appropriately displayed to the user, variousproblems will occur. For example, in the system using an HMD, theline-of-sight direction of the user changes in various manners and thereexists a large number of objects in the VR space. Therefore, when thedisplay of information to the user is not appropriate, the screen maybecome hard to see or the virtual reality for the user may bedeteriorated.

BRIEF DESCRIPTION I/F THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of asimulation system according to the present embodiment.

FIG. 2A and FIG. 2B illustrate an example of an HMD used in the presentembodiment.

FIG. 3A and FIG. 3B illustrate another example of the HMD used in thepresent embodiment.

FIG. 4 is a diagram illustrating an example of a game image generated inthe present embodiment.

FIG. 5 is a diagram illustrating an example of a game image generated inthe present embodiment.

FIG. 6 is a diagram illustrating an example of a game image generated inthe present embodiment.

FIG. 7 is a drawing describing display of status of a character.

FIG. 8A to FIG. 8D are diagrams describing a display method ofinformation display objects.

FIG. 9A and FIG. 9B are diagrams describing a method for changing adisplay mode in the present embodiment.

FIG. 10A and FIG. 10B are diagrams describing a method for changing thedisplay mode in the present embodiment.

FIG. 11A and FIG. 11B are diagrams describing a method for changing thedisplay mode in the present embodiment.

FIG. 12A and FIG. 12B are diagrams describing a method for changing thedisplay mode in the present embodiment.

FIG. 13A and FIG. 13B are diagrams describing a method for changing thedisplay mode according to the types, priorities, and importance levelsof the information display objects.

FIG. 14 is a diagram describing a method for changing the display modeaccording to the lengths of time elapsed from display of the informationdisplay objects.

FIG. 15A and FIG. 15B are diagrams describing a method for determining achange condition using a change angle, a change speed, or a changeacceleration.

FIG. 16 is a diagram describing a method for differentiating the contentof the change process of the display mode according to a change angle, achange speed, or a change acceleration.

FIG. 17A and FIG. 17B are diagrams describing a method for determinationon a change condition using a determination range.

FIG. 18 is a diagram describing a method for setting the determinationranges of a plurality of information display objects.

FIG. 19 is a diagram describing a method for performing the changeprocess of the display mode on the information display object in thecentral part.

FIG. 20 is a diagram describing a method for differentiating the contentof the change condition or the change process according to theinformation display objects.

FIG. 21A and FIG. 21B are diagrams describing a method for performingthe change process of the display mode according to depth values of theinformation display objects.

FIG. 22A to FIG. 22C are diagrams describing a method for changing thedisplay mode according to the length of a time elapsed since the changecondition is satisfied.

FIG. 23A to FIG. 23C are diagrams describing a return process of thechange process of the display mode.

FIG. 24A to FIG. 24C are diagrams describing a return process of thechange process of the display mode.

FIG. 25 is a flowchart illustrating a detailed example of the returnprocess of the change process of the display mode.

FIG. 26A and FIG. 26B are diagrams describing a method fordifferentiating brightness of a display image between when the changeprocess of the display mode is not executed and when the change processof the display mode is executed.

FIG. 27A and FIG. 27B are diagrams describing an activation process ofan effect of a command according to positional relationship informationbetween the user and the character.

FIG. 28A and FIG. 28B are diagrams describing an activation process ofan effect of a command according to positional relationship informationbetween the user and the character.

FIG. 29A and FIG. 29B are diagrams describing an activation process ofan effect of command according to line-of-sight relationship informationbetween the user and the character.

FIG. 30 is a diagram describing an activation process of an effect of acommand according to gaze information of the user.

FIG. 31 is a flowchart illustrating a detailed process example accordingto the present embodiment.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following disclosure provides various different embodiments andexamples to embody different features of the presented subject matters.These are apparently provided for illustrative purposes only and are notintended to be construed in a limiting sense. The present disclosure mayinclude repeated use of reference numerals and/or characters in variousexamples. Such repetitions are provided for a concise and cleardescription, and do not simply require any relation with variousembodiments and/or configurations described. Furthermore, a descriptionof a first element “connected” or “coupled” to a second element includessome embodiments in which the first element is directly connected orcoupled to the second element and other embodiments in which the firstelement is indirectly connected or coupled to the second element withone or more other elements interposed therebetween.

In accordance with one of some embodiments, it is possible to provide asimulation system that implements an appropriate display process ofinformation display objects in a system using a head mounted display, aprocessing method, and an information storage medium, and the like.

One of some embodiments relates to a simulation system comprising aprocessor including hardware. The processor performs: a virtual spacesetting process of setting a virtual space in which a plurality ofobjects are arranged; a virtual camera control process of controlling avirtual camera corresponding to a point-of-view of a user wearing a headmounted display; and a display process of generating an image as viewedfrom the virtual camera in the virtual space as a display image on thehead mounted display. In the virtual space setting process, theprocessor arranges at least one information display object in aline-of-sight direction of the user. In the display process, when it isdetermined that a given change condition is satisfied by a change in thepoint-of-view of the user, the processor performs a change process of adisplay mode of the information display object.

In accordance with one of some embodiments, the setting process of thevirtual space in which the plurality of objects are arranged isperformed and the image as viewed from the virtual camera correspondingto the point-of-view of the user is generated as a display image on thehead mounted display. Then, when the information display is arranged inthe line-of-sight direction of the user and it is determined that thegiven change condition is satisfied by a change in the point-of-view ofthe user (a change in at least one of the line-of-sight direction andthe point-of-view position), the process of changing the display mode ofthe information display object is performed. Accordingly, when thechange condition for the information display subject arranged in theline-of-sight direction of the user is satisfied, the display mode ofthe information display object is changed to display a more appropriateimage on the head mounted display. Therefore, it is possible to providethe simulation system that implements the appropriate display process ofthe information display object in the system using the head mounteddisplay.

In accordance with one of some embodiments, in the virtual space settingprocess, the processor may arrange the information display objectcontrolled to follow a change in the point-of-view of the user in thevirtual space.

Accordingly, when the line-of-sight direction or point-of-view positionof the user changes, the information display object is controlled tofollow the change, thereby making it possible to implement presentationof appropriate information by the information display object.

In accordance with one of some embodiments, in the display process, theprocessor may perform the change process of the display mode accordingto the type, priority, or importance level of the information displayobject.

Accordingly, it is possible to implement the appropriate change processof the display mode according to the type, priority, or importance levelof the information display object.

In accordance with one of some embodiments, in the display process, theprocessor may perform the change process of the display mode accordingto the length of a time elapsed from the display of the informationdisplay object.

Accordingly, it is possible to implement the change process of thedisplay mode reflecting the length of a time elapsed from the display ofthe information display object.

In accordance with one of some embodiments, as the change process of thedisplay mode, the processor may perform an erase process of theinformation display object, a semi-transparentization process of theinformation display object, a cancel process of following by theinformation display object to the line-of-sight direction of the user, amovement process of the information display object, a size changeprocess of the information display object, or a color information changeprocess of the information display object.

Accordingly, when the change condition is satisfied, it is possible toimplement the appropriate display process of the information displayobject by performing the erase process or semi-transparentizationprocess of the information display object, the cancel process offollowing by the information display object, the movement process andsize change process of the information display object, or the colorinformation change process of the information display object.

In accordance with one of some embodiments, the processor may perform adetermination process of determining whether the given change conditionis satisfied, and in the determination process, the processor maydetermine whether the given change condition is satisfied based onpoint-of-view change information of the user.

Accordingly, when a change in the point-of-view of the user is large orabrupt, it is determined that the change condition is satisfied so thatthe change process of the display mode of the information display objectcan be performed.

In accordance with one of some embodiments, the processor may perform adetermination process of determining whether the given change conditionis satisfied. A determination range is set to the information displayobject. In the determination process, when a gaze position of the userin the line-of-sight direction exceeds the determination range, theprocessor may determine that the given change condition is satisfied.

Accordingly, it is possible to use the information display object as areference to detect a relative change in the line-of-sight direction tothe information display object, thereby determining the change conditionfor the change process of the display mode.

In accordance with one of some embodiments, as the determination range,a first determination range may be set to a first information displayobject, and as the determination range, a second determination rangedifferent from the first determination range may be set to a secondinformation display object.

Accordingly, it is possible to set different determination rangesaccording to information display objects to determine the changecondition based on different determination criteria.

In accordance with one of some embodiments, when a plurality of theinformation display objects are arranged in the virtual space, in thedisplay process, the processor may perform the change process of thedisplay mode on the information display object positioned in a centralpart as seen from the point-of-view of the user.

Accordingly, it is possible to prevent a situation in which theinformation display object positioned in the central part as seen fromthe point-of-view of the user blocks the field of view of the user.

In accordance with one of some embodiments, when a plurality of theinformation display objects are arranged in the virtual space, in thedisplay process, the processor may differentiate content of the givenchange condition or content of the change process of the display modebetween the first information display object and the second informationdisplay object.

Accordingly, it is possible to set the appropriate content of the changecondition or the change process according to information display objectsto execute the change process of the display mode.

In accordance with one of some embodiments, in the display process, theprocessor may perform the change process of the display mode accordingto a depth value of the information display object.

Accordingly, it is possible to implement the change process of thedisplay mode of the information display object using effectively thedepth value of the information display object.

In accordance with one of some embodiments, in the display process, theprocessor may perform the change process of the display mode accordingto length of a time elapsed since the given change condition issatisfied.

Accordingly, it is possible to implement the change process of thedisplay mode reflecting the length of a time elapsed since the changecondition is satisfied.

In accordance with one of some embodiments, the processor may perform adetermination process of determining whether the given change conditionis satisfied. In the display process, when it is determined that thegiven change condition is satisfied by the determination process, theprocessor may perform the change process of the display mode of theinformation display object. After the change process of the display modeof the information display object, when it is determined in thedetermination process that a given return condition is satisfied, theprocessor may perform a return process of the change process of thedisplay mode of the information display object.

Accordingly, when the change condition is satisfied and the changeprocess of the display mode of the information display object isperformed, it is possible to prevent the display mode of the informationdisplay object from remaining in the changed state.

In accordance with one of some embodiments, in the determinationprocess, when a given time has elapsed after the change process of thedisplay mode, or when no change in the line-of-sight direction of theuser has been detected during a given time after the change process ofthe display mode, or when gaze of the user on a given object has beendetected after the change process of the display mode, the processor maydetermine that the given return condition is satisfied.

Accordingly, it is possible to determine appropriate return conditionscorresponding to various situations to execute the return process.

In accordance with one of some embodiments, in the display process, theprocessor may differentiate brightness of the display image on the headmounted display between when the change process of the display mode ofthe information display object is not performed and when the changeprocess of the display mode of the information display object isperformed.

Accordingly, it is possible to set the brightness of the display imagedepending on whether the change process of the display mode isperformed, thereby generating a more appropriate display image.

One of some embodiments relates to a processing method. The methodincludes: a virtual space setting process of setting a virtual space inwhich a plurality of objects are arranged; a virtual camera controlprocess of controlling a virtual camera corresponding to a point-of-viewof a user wearing a head mounted display; and a display process ofgenerating an image as viewed from the virtual camera in the virtualspace as a display image on the head mounted display. In the virtualspace setting process, at least one information display object isarranged in a line-of-sight direction of the user. In the displayprocess, when it is determined that a given change condition issatisfied by a change in the point-of-view of the user, a change processof a display mode of the information display object is performed.

Exemplary embodiments are described below. Note that the followingexemplary embodiments do not in any way limit the scope of the contentdefined by the claims laid out herein. Note also that all of theelements described in the present embodiment should not necessarily betaken as essential elements.

1. Simulation System

FIG. 1 is a block diagram illustrating a configuration example of asimulation system (a simulator, a game system) according to the presentembodiment. The simulation system according to the present embodiment isa system that simulates Virtual Reality (VR) for example, and can beapplied to various systems such as a game system providing gamecontents, a real-time simulation system including a sports eventsimulator and a driving simulator, a content providing system thatprovides a content such as a movie, or an operating system forimplementing a remote controlled operation. The simulation systemaccording to the present embodiment is not limited to the configurationillustrated in FIG. 1, and can be modified in various ways includingomitting some of its components (sections) or adding another component.

An image capturing section 150 includes one or more cameras. Each of thecameras is formed from an optical system including a lens (a wide-anglelens) and imaging elements such as a CCD and a CMOS sensor. The imagecapturing section 150 may be provided with one or more microphones.Using the image capturing section 150 makes it possible to detectinformation on a user's motion (motion information on individual bodyparts and skeleton information) and implement user recognitionprocessing (face recognition and the like). In addition, providing theimage capturing section 150 with a plurality of cameras makes itpossible to recognize the depth direction in a space and determine thefront-back positional relationship between two users playing in front ofthe image capturing section 150. Further, providing the image capturingsection 150 with a plurality of microphones makes it possible to detectthe direction of a sound source and the like. For example, using theimage capturing section 150 with a plurality of cameras and a pluralityof microphones makes it possible to implement a game that can beintuitively played using the user's motion and voice as operationinformation.

An operation section 160 is used by a user (player) to input varioustypes of operation information (input information). The operationsection 160 can be implemented by various operation devices such as anoperation button, a direction designating key, a joystick, a handle, apedal, and a lever for example.

A storage section 170 stores therein various types of information. Thestorage section 170 functions as a work area for a processing section100, a communication section 196, and the like. The storage section 170stores therein a game program and game data required for executing thegame program. The function of the storage section 170 can be implementedby a semiconductor memory (dynamic random access memory (DRAM), videoRAM (VRAM)), a hard disk drive (HDD), a solid state drive (SSD), anoptical disc device, or the like. The storage section 170 includes anobject information storage section 172 and a rendering buffer 178.

An information storage medium 180 (computer-readable medium) storestherein a program, data, and the like. The function of the informationstorage medium 180 can be implemented by an optical disc (a digitalversatile disc (DVD), a Blu-ray disc (BD), a compact disc (CD)), an HDD,a semiconductor memory (read only memory (ROM)), and the like. Theprocessing section 100 performs various processes according to thepresent embodiment based on a program (data) stored in the informationstorage medium 180. Thus, the information storage medium 180 storestherein a program for causing a computer (a device including an inputdevice, the processing section, the storage section, and an outputsection) to function as the sections according to the present embodiment(a program for causing a computer to perform processes of the sections).

A head mounted display (HMD) 200 is a device that is worn on the head ofthe user, and displays an image in front of the eyes of the user. TheHMD 200 is preferably a non-transparent type, but may also be atransparent type. The HMD 200 may be what can be referred to as aneye-piece type HMD.

The HMD 200 includes a sensor section 210, a display section 220, and aprocessing section 240. The sensor section 210 implements a trackingprocess such as head tracking for example. For example, the position andthe direction of the HMD 200 are identified through the tracking processperformed with the sensor section 210. With the position and thedirection of the HMD 200 thus identified, a point-of-view position and aline-of-sight direction of the user as point-of-view information can beidentified.

Various tracking schemes can be employed. For a first tracking scheme asan example of the tracking scheme, a plurality of light emittingelements (LEDs) are provided as the sensor section 210, as will bedescribed in detail later with reference to FIG. 2A and FIG. 2B. Thelight from the plurality of light emitting elements is imaged by theimage capturing section 150 to identify the position and the directionof the HMD 200 (the head of the user) in a three-dimensional space ofthe real world. For a second tracking scheme, a plurality of lightreceiving elements (such as photodiodes) are provided as the sensorsection 210, as will be described in detail later with reference to FIG.3A and FIG. 3B. With the plurality of light receiving elements receivinglight (such as a laser beam) from a light emitting element (such asLEDs) provided outside, the position and the direction of the HMD 200are identified. A third tracking scheme uses a motion sensor, providedto the sensor section 210, to identify the position and the direction ofthe HMD 200. For example, the motion sensor can be implemented with anacceleration sensor, a gyro sensor, or the like. For example, theposition and the direction of the HMD 200 in the three-dimensional spacein the real world can be identified with a 6-axis motion sensorincluding a 3-axis acceleration sensor and a 3-axis gyro sensor. Theposition and the direction of the HMD 200 may be identified with acombination of the first tracking scheme and the second tracking scheme,or a combination of the first tracking scheme and the third trackingscheme. A tracking process of directly identifying the point-of-viewposition and line-of-sight direction of the user, instead of identifyingthe position and the direction of the HMD 200 to identify thepoint-of-view position and line-of-sight direction of the user, may beemployed.

For example, the display section 220 of the HMD 200 can be implementedwith an organic electroluminescence display (OEL), a liquid crystaldisplay (LCD), or the like. For example, the display section 220 of theHMD 200 is provided with a first display or a first display area set tobe in front of the left eye of the user, and a second display or asecond display area set to be in front of the right eye of the user,whereby stereoscopic view can be provided. The stereoscopic view isimplemented with left-eye and right-eye images, with parallax, generatedto be respectively displayed on the first and the second displays.Alternatively, the left-eye image and the right-eye image arerespectively displayed on the first and the second display areas of asingle display. The HMD 200 is provided with two eyepieces (fish-eyelenses) for the left-eye and the right-eye so that a VR space can beprovided entirely over the field of view of the user. A correctionprocess is performed for the left-eye image and the right-eye image tocorrect distortion produced in an optical system such as the eyepiece.This correction process is performed by the display processing section120.

The processing section 240 of the HMD 200 performs various processesrequired in the HMD 200. For example, the processing section 240performs a control process for the sensor section 210, a display controlprocess for the display section 220, or the like. The processing section240 may perform a three-dimensional acoustic (stereophonic sound)process to simulate direction, distance and spreading of sound in threedimensions.

A sound output section 192 outputs sound generated in accordance withthe present embodiment, and can be implemented by a speaker, aheadphone, or the like.

An interface (I/F) section 194 performs an interface process for aportable information storage medium 195. The function of the I/F section194 can be implemented with an application specific integrated circuit(ASIC) for the I/F process. The portable information storage medium 195is a storage device that stores therein various types of informationfrom the user, and holds the information without power supply. Theportable information storage medium 195 can be implemented with anintegrated circuit (IC) card (memory card), a universal serial bus (USB)memory, a magnetic card, or the like.

The communication section 196 communicates with external apparatuses(other devices) through a wired or wireless network. The function of thecommunication section 196 can be implemented with a communication ASIC,hardware such as a communication processor, or a communication firmware.

The program (data) for causing a computer to function as the sectionsaccording to the present embodiment may be distributed to theinformation storage medium 180 (or the storage section 170) from aninformation storage medium of a server (host device) through a networkand the communication section 196. The scope of the present disclosurecan include such a configuration where the information storage medium ofthe server (host device) is used.

The processing section 100 (processor) performs a game process(simulation process), a moving body process, a virtual camera controlprocess, a display process, or sound process based on operationinformation from the operation section 160, tracking information aboutthe HMD 200 (information about at least one of the position anddirection of the HMD, information about at least one of thepoint-of-view position and the line-of-sight direction), a program, andthe like.

Processes (functions) according to the present embodiment performed bysections of the processing section 100 can be implemented by a processor(processor including hardware). For example, the processes according tothe present embodiment can be implemented by a processor that operatesbased on information such as a program and a memory that stores thereinthe information such as the program. For example, the processor mayimplement the functions of the sections in discrete hardware or inintegrated hardware. For example, the processor may include hardware,and the hardware may include at least one of a circuit that processes adigital signal and a circuit that processes an analog signal. Forexample, the processor may include one or a plurality of circuit devices(such as an integrated circuit (IC) for example) or one or a pluralityof circuit elements (such as a resistor and a capacitor for example)mounted on a circuit board. For example, the processor may be a centralprocessing unit (CPU). However, the processor is not limited to the CPU,and various processors such as a graphics processing unit (GPU) or adigital signal processor (DSP) may be used. The processor may be ahardware circuit such as an ASIC. The processor may include an amplifiercircuit, a filter circuit, or the like that processes an analog signal.The memory (storage section 170) may be a semiconductor memory such as astatic random access memory (SRAM) and a DRAM or may be a resistor.Furthermore, the memory may be a magnetic storage device such as a harddisk device (HDD) or may be an optical storage device such as an opticaldisc device. For example, the memory stores therein a computer-readablecommand, and the processes (functions) of the sections of the processingsection 100 are implemented with the processor executing the command.This command may be a set of commands forming a program, or may be acommand for instructing an operation to a hardware circuit of theprocessor.

The processing section 100 includes an input processing section 102, acalculation processing section 110, and an output processing section140. The calculation processing section 110 includes a game processingsection 112, a moving body processing section 114, a determinationsection 115, a virtual space setting section 116, a virtual cameracontrol section 118, a display processing section 120, and a soundprocessing section 130. As described above, the processes according tothe present embodiment performed by these sections may be implemented bya processor (or a processor and a memory). Various modifications may bemade with some of these components (sections) omitted, or anothercomponent added.

The input processing section 102 performs an input process including: aprocess of receiving operation information or tracking information; aprocess of reading information from the storage section 170; and aprocess of receiving information through the communication section 196.For example, the input processing section 102 performs an input processincluding: a process of acquiring operation information input by a userby using the operation section 160 and tracking information detected bythe sensor section 210 of the HMD 200; a process of reading information,designated with a read command, from the storage section 170; and aprocess of receiving information from an external apparatus (such as aserver) through a network. The receiving process includes a process ofinstructing the communication section 196 to receive information,acquiring the information received by the communication section 196, andwriting the information to the storage section 170.

The calculation processing section 110 performs various calculationprocesses. For example, the calculation processes are performed for agame process (simulation process), a moving body process, a virtualcamera control process, a display process, a sound process, or the like.

The game processing section 112 (a program module for a game process)performs various game processes for the user to play the game. In otherwords, the game processing section 112 (simulation processing section)performs various simulation processes to enable the user to experiencevirtual reality. Examples of the game process include a process ofstarting the game when a game start condition is satisfied, a process ofmaking the started game progress, a process of ending the game when agame end condition is satisfied, and a process of calculating a gameresult.

The moving body processing section 114 (a program module for a movingbody process) performs various processes for a moving body that moves ina virtual space. For example, a process of moving the moving body in avirtual space (object space or game space) or a process of causing themoving body to make an action is performed. For example, the moving bodyprocessing section 114 performs a control process based on the operationinformation input by the user using the operation section 160, trackinginformation acquired, a program (movement/operation algorithm), andvarious types of data (motion data), and the like. The control processincludes moving the moving body (model object) in the virtual space andcausing the moving body to make an action (motion, animation).Specifically, a simulation process is performed to sequentially obtainmovement information (position, rotational angle, speed, oracceleration) and action information (a position and a rotational angleof a part object) of the moving body on a frame (for example, 1/60seconds) by frame basis. The frame is a unit of time for performing amovement/action process (simulation process) of the moving body and animage generation process. For example, the moving body is a virtual user(virtual player) in a virtual space corresponding to the user (player)in the real space, a ridden moving body (operated moving body) ridden(operated) by the virtual user, or a character as a target of play bythe virtual user.

The determination section 115 (a program module for determinationprocesses) performs various determination processes. The determinationsection 115 will be described later in detail.

The virtual space setting section 116 (a program module for a virtualspace setting process) performs a setting process for a virtual space(object space) where a plurality of objects is arranged. For example, aprocess of setting an arrangement of various objects (an object formedby a primitive surface such as a polygon, a free-form surface or asubdivision surface) representing various display objects such as amoving body (such as a person, a robot, a car, a train, an aircraft, aboat, a monster, or an animal), a map (terrain), a building, audienceseats, a course (road), woods, a wall, and a water surface in thevirtual space is performed. Specifically, a position and a rotationalangle (that is the same as an orientation or a direction) of an objectin a world coordinate system are determined, and the object is arrangedat the position (X, Y, Z) at the rotational angle (rotational anglesabout X, Y, and Z axes). Thus, the object information storage section172 of the storage section 170 stores object information in associationwith an object number. The object information includes information abouta position, rotational angle, a movement speed, a moving direction, andthe like of an object (part object) in the virtual space. Examples ofthe process performed by the virtual space setting section 116 includeupdating the object information on a frame by frame basis.

The virtual camera control section 118 (a program module for a virtualcamera control process) performs a control process for a virtual camera(point-of-view, reference virtual camera) to generate an image as viewedfrom a given (any) point-of-view in the virtual space. For example, aprocess of controlling a point-of-view position or a line-of-sightdirection (position or orientation) of the virtual camera is performed.Specifically, a process (a process of controlling a point-of-viewposition, a line-of-sight direction, or an angle of view) of controllingthe position (X, Y, Z) of the virtual camera and a rotational angle (arotational angle about an X, Y, or Z axis) as orientation information isperformed. The virtual camera corresponds to a point-of-view of the user(virtual user). When stereoscopic view is implemented, a left-eye firstpoint-of-view (left-eye first virtual camera) and a right-eye secondpoint-of-view (right-eye second virtual camera) are set.

The display processing section 120 (a program module for a displayprocess) performs a display process for a game image (simulation image).For example, a rendering process is performed based on results ofvarious processes (a game process, a simulation process) performed bythe processing section 100 to generate an image, and the image isdisplayed on the display section 220 of the HMD 200. Specifically, ageometry process such as coordinate transformation (world coordinatetransformation, camera coordinate transformation), a clipping process, aperspective transformation, or a light source process is performed.Rendering data (coordinates of the vertex position of the primitivesurface, texture coordinates, color data, a normal vector, an a value,or the like) is generated based on a result of the process. An object(one or a plurality of primitive surfaces) after the perspectivetransformation (after the geometry process) is rendered in the renderingbuffer 178 (a frame buffer, a work buffer or the like that can storeimage information in a unit of pixels), based on the rendering data(primitive surface data). Thus, an image as viewed from the virtualcamera (a given point-of-view, a left-eye, right-eye, first, or secondpoint-of-view) is generated in the virtual space. The rendering processperformed by the display processing section 120 can be implemented witha vertex shader process, a pixel shader process, or the like.

The sound processing section 130 (a program module for sound process)performs a sound process based on a result of various processesperformed by the processing section 100. Specifically, game sound suchas a song (music, background music (BGM)), a sound effect, or a voice isgenerated to be output by the sound output section 192. A part(three-dimensional acoustic process for example) of the sound processperformed by the sound processing section 130 may be implemented by theprocessing section 240 of the HMD 200.

The output processing section 140 performs an output process ofoutputting various types of information. For example, the outputprocessing section 140 performs the output process including: a processof writing information to the storage section 170; and a process oftransmitting information through the communication section 196. Forexample, the output processing section 140 performs a process of writinginformation, designated by a write command, to the storage section 170,and a process of transmitting information to an external apparatus (suchas a server) through a network. This transmission process is a processof instructing the communication section 196 to transmit information,and a process of designating the information to be transmitted to thecommunication section 196.

The simulation system of the present embodiment includes the virtualspace setting section 116, the virtual camera control section 118, andthe display processing section 120 as illustrated in FIG. 1. The virtualspace setting section 116 performs a setting process of a virtual spacein which a plurality of objects is arranged. For example, the virtualspace setting section 116 performs a process of setting the positionsand directions of a plurality of objects and arranging the same in avirtual space. The objects are display objects that are game elementsappearing in a game or constituting a game space.

The virtual camera control section 118 performs a control process of thevirtual camera corresponding to the point-of-view of the user wearingthe HMD 200. For example, the virtual camera control section 118controls the virtual camera set as a first person point-of-view of theuser. For example, the virtual camera is set to be at a positioncorresponding to the point-of-view of the virtual user moving in thevirtual space, and the point-of-view position and the line-of-sightdirection of the virtual camera are set to control the position(position coordinates) and the orientation (a rotational angle about arotation axis) of the virtual camera.

The display processing section 120 generates an image as viewed from thevirtual camera (user point-of-view) in the virtual space as a displayimage (display video) of the HMD 200. For example, an image as viewedfrom a given point-of-view in the object space as the virtual space isgenerated. The generated image is a stereoscopic image, for example.

In the present embodiment, the virtual space setting section 116arranges at least one information display object in the line-of-sightdirection of the user (a field of view range). For example, the virtualspace setting section 116 arranges an information display object in thefield of view range (the front direction) of the user. Specifically, thevirtual space setting section 116 arranges an information display objectsuch that the information display object can be seen in the frontdirection of the user.

When it is determined that a given change condition is satisfied by achange in the point-of-view of the user, the display processing section120 performs a change process of a display mode (display method ordisplay scheme) of the information display object. The change in thepoint-of-view refers to a change in at least one of the line-of-sightdirection and the point-of-view position. For example, when it isdetermined that the change condition is satisfied by the user greatlychanging the line-of-sight direction or the point-of-view position (userposition), the display processing section 120 performs the process ofchanging the display mode of the information display object. Forexample, when the change condition is satisfied, the display processingsection 120 generates a display image for the HMD 200 such that theimage is differently seen from the image when the change condition isnot satisfied, and displays the image on the HMD 200.

That is, the information display object is arranged in the virtualspace, for example, in the line-of-sight direction of the user asdescribed above. For example, the information display object is arrangedin the line-of-sight direction (front direction or field of view range)of the user. Therefore, when the line-of-sight direction orpoint-of-view position of the user changes greatly, if the informationdisplay object is displayed in the display mode before the change of theline-of-sight direction or the point-of-view position, an inappropriatedisplay image may be displayed on the HMD 200. For example, when theline-of-sight direction of the user is oriented to the front direction,displaying the information display object in that direction makes itpossible to implement appropriate information transmission by theinformation display object. On the other hand, in the system using theHMD 200, the vast VR space spreads over the entire field of view of theuser. Accordingly, the user may move the line-of-sight direction notonly to the front direction but also to all the directions in such amanner as to look around the space or may move to various directions. Inthis case, if the information display object remains displayed in thedisplay mode in which the line-of-sight direction is oriented to thefront direction or the point-of-view position (user position) isstationary, the display may be inappropriate for the user. Accordingly,when it is determined that the given change condition is satisfied by achange in the point-of-view of the user, the display processing section120 performs the process of changing the display mode of the informationdisplay object to prevent such inappropriate display.

The virtual space setting section 116 may arrange the informationdisplay object (at least one information display object) controlled tofollow a change in the point-of-view of the user (a change in thepoint-of-view of the virtual camera) in the virtual space. Specifically,the virtual space setting section 116 arranges the information displayobject controlled to follow at least one of the line-of-sight directionand the point-of-view position of the user. For example, when theline-of-sight direction or the point-of-view position of the userchanges, the arrangement position (three-dimensional position) of theinformation display object in the virtual space also changesaccordingly. For example, when the line-of-sight direction of the userchanges leftward, rightward, upward, or downward, the position anddirection of the information display object in the virtual space alsochange to follow the change in that direction. Otherwise, when thepoint-of-view position of the user (user position) moves leftward,rightward, upward, or downward, the position and others of theinformation display object in the virtual space also change to followthat movement (position change). For example, when the informationdisplay object is formed from a billboard polygon, the billboard polygonconstituting the information display object is arranged to face thepoint-of-view of the user (orthogonal to the line-of-sight direction ofthe user).

In the case where the information display object is controlled to followa change in the point-of-view of the user, when the line-of-sightdirection or point-of-view position of the user changes greatly, theposition of the information display object also changes greatlyfollowing that change, which leads to a situation in which a displayimage cannot be appropriately displayed. Also in this case, in thepresent embodiment, the display processing section 120 performs theprocess of changing the display mode of the information display objectto prevent the occurrence of such a situation.

The virtual space setting section 116 also arranges the informationdisplay object to follow a change in the point-of-view at a given depthposition in the line-of-sight direction of the user. For example, thevirtual space setting section 116 arranges the information displayobject to follow a change in the point-of-view (a change in theline-of-sight direction or the point-of-view position) such that thereis no change in the depth position (depth distance) of the informationdisplay object to the point-of-view of the user (virtual camera). Forexample, the virtual space setting section 116 arranges the informationdisplay object formed from a billboard polygon, for example, at a givendepth position (a position in a Z coordinate axis) from thepoint-of-view of the user (virtual camera).

In addition, the display processing section 120 may perform the changeprocess of the display mode according to the type, priority, orimportance level of the information display object. For example, thedisplay processing section 120 performs the change process of thedisplay mode of the information display object reflecting the type,priority, or importance level of the information display object. Thetype of the information display object is determined by the type ofinformation provided by the information display object or the like, forexample. For example, the storage section 170 stores information on thetype in association with each information display object, and theinformation on the type is used to perform the change process of thedisplay mode. The priority and importance level of the informationdisplay object are information indicating how high the informationprovided by the information display object is in priority and importancelevel. For example, the storage section 170 stores information on thepriority and importance level in association with each informationdisplay object, and the information on the priority and importance levelis used to perform the change process of the display mode.

For example, when the information display object is of a first type (forexample, an information display object for command selection), thedisplay processing section 120 does not perform the change process ofthe display mode of the information display object. On the other hand,when the information display object is of a second type (for example, aninformation display object for description display), the displayprocessing section 120 performs the change process of the display modeof the information display object (erasing, semi-transparentization, orthe like). Whether the information display object is of the first typeor of the second type is stored in the storage section 170 inassociation with the information display object.

The display processing section 120 also performs the change process ofthe display mode using the priority or the importance level inassociation with the information display object. For example, thestorage section 170 stores the information on the priority and theimportance level in association with the information display object. Inthe case of performing the change process of the display mode, thedisplay processing section 120 reads the information on the priority andthe importance level associated with the information display object tobe processed from the storage section 170 and performs the changeprocess of the display mode of the information display object.

The display processing section 120 also performs the change process ofthe display mode according to the length of a time elapsed from thedisplay of the information display object. For example, the displayprocessing section 120 measures and records the length of a time elapsedfrom the display of each information display object. Then, in the caseof performing the change process of the display mode of the informationdisplay object, the display processing section 120 performs the processreflecting the elapsed time. For example, the display processing section120 performs the change process of the display mode of the informationdisplay object with a longer time elapsed from the display in priorityto the information display object with a shorter time elapsed from thedisplay.

The display processing section 120 also performs, as the change processof the display mode, an erase process of the information display object,a semi-transparentization process of the information display object, acancel process of following by the information display object to thepoint-of-view of the user, a movement process of the information displayobject, a size change process of the information display object, a colorinformation change process of the information display object, and thelike.

The erase process of the information display object can be implementedby excluding the information display object from display targets(display list) or semi-transparentizing the information display object.The semi-transparentization process of the information display objectcan be implemented by changing the semi-transparency (a value) of theinformation display object such that the information display objectbecomes more semi-transparent, for example. Although the informationdisplay object is controlled to follow a change in the point-of-view ofthe user as described above, cancelling (reducing) this follow controlmakes it possible to implement the cancel process of following by theinformation display object to a change in the point-of-view of the user.The movement process of the information display object can beimplemented by moving the information display object not so as to bedisplayed in the line-of-sight direction of the user, for example. Thesize change process of the information display object can be implementedby reducing the size of the information display object to be lessoutstanding in the line-of-sight direction of the user, for example. Thecolor information change process of the information display object canbe implemented by changing the brightness, chroma, or hue of theinformation display object, for example. For example, the colorinformation change process of the information display object may beperformed by bringing the color information of the information displayobject closer to the color information of the background, for example,such that the information display object blends into the background.This process can be implemented by a blending or depth queuing based onthe color information of the information display object and the colorinformation of the background, for example.

The information display object is a display object for descriptiondisplay, a display object for status display, a display object forconversation display, a display object for command selection, or adisplay object for menu display, for example. The display object fordescription display is a display object for providing the user withvarious descriptions on a game or the like, for example. The displayobject for status display is a display object for informing the userabout various situations in a game or the like, for example. The displayobject for conversation display is a display object for informing theuser about the contents of various conversations, for example. Forexample, the display object for conversation display is a display objectfor informing the contents of the conversations of a character as atarget of play by the user. The display object for command selection isa display object used by the user to select a command in a game or thelike, for example. For example, the display object for command selectionis a display object for presenting command options to the user, forexample. The display object for menu display is a display object forpresenting a menu screen in a game or the like to the user. For example,the display object for menu display is a display object for presenting amenu screen for game mode, game setting (option setting), or game endinstruction to the user. The information display object is a displayobject for providing the user with information by using characters,icons, or graphics, for example. As an example, the information displayobject can be implemented by a billboard polygon in which informationfor the user is rendered.

Besides the information display object, a plurality of objects arearranged in the virtual space. Examples of objects other than theinformation display object include a moving body, a fixed body, an item,or a background object, which appears in a game. The moving bodyappearing in a game is a character as a target of play by the user or adisplay object moving in various directions and to various positions ina game, for example. The fixed body is a display object that does notmove like a moving body but is fixed at a predetermined position in thevirtual space. The item is a display object such as various kinds oftools appearing in a game, which is a display object to be played, held,or gripped by the user. The object of the background is an objectconstituting the background of a game or the like.

The simulation system also includes the determination section 115 thatdetermines whether a given change condition is satisfied. When thedetermination section 115 determines that the given change condition issatisfied, the display processing section 120 performs the changeprocess of the display mode of the information display object.

The determination section 115 determines whether the given changecondition is satisfied based on point-of-view change information of theuser, for example. The point-of-view change information is line-of-sightdirection change information or point-of-view position changeinformation, for example. The line-of-sight direction change informationis a change angle of the line-of-sight direction, a change speed of theline-of-sight direction, or a change acceleration of the line-of-sightdirection, for example. The point-of-view position change information isa change distance of the point-of-view position (displacement), a changespeed of the point-of-view position, a change acceleration of thepoint-of-view position, or the like. For example, when the change angleof the line-of-sight direction of the user becomes equal to or largerthan a given angle, when the change speed of the line-of-sight directionof the user becomes equal to or higher than a given speed, or when thechange acceleration of the line-of-sight direction of the user becomesequal to or higher than a given acceleration, the determination section115 determines that the given change condition is satisfied. Otherwise,when the change distance of the point-of-view position of the userbecomes equal to or longer than a given distance, when the change speedof the point-of-view position of the user becomes equal to or higherthan a given speed, or when the change acceleration of the point-of-viewposition of the user becomes equal to or higher than a givenacceleration, the determination section 115 determines that the givenchange condition is satisfied. When the change condition is satisfied inthis manner, the display processing section 120 performs the changeprocess of the display mode of the information display object.

The change angle in the line-of-sight direction of the user is a changeangle of vector of the line-of-sight direction, for example. Forexample, when an angle indicating the direction of the vector of theline-of-sight direction (for example, an angle with respect to apredetermined reference axis) changes from a first angle to a secondangle, the difference between the first angle and the second angle isthe change angle. The change speed of the line-of-sight direction refersto the amount of a change in the angle of the vector of theline-of-sight direction per predetermined unit time (for example, oneframe), for example, which is equivalent to an angular speed. The changeacceleration of the line-of-sight direction refers to the amount of achange in the change speed of angle of the vector of the line-of-sightdirection per predetermined unit time, for example, which is equivalentto an angular acceleration. The change distance of the point-of-viewposition of the user refers to, when the point-of-view position of theuser changes from a first position to a second position, a distancebetween the first position and the second position, for example. Thechange speed of the point-of-view position of the user refers to theamount of a change of the point-of-view position per predetermined unittime. The change acceleration of the point-of-view position of the userrefers to the amount of a change in the change speed of thepoint-of-view position per predetermined unit time.

The display processing section 120 differentiates the content of thechange process of the display mode of the information display objectaccording to the point-of-view change information of the user. Forexample, the display processing section 120 changes the content of thechange process of the display mode of the information display objectaccording to the change angle, the change speed, or the changeacceleration of the line-of-sight direction of the user. Otherwise, thedisplay processing section 120 differentiates the content of the changeprocess of the display mode of the information display object accordingto the change distance, the change speed, or the change acceleration ofthe point-of-view position of the user. For example, the displayprocessing section 120 changes the contents of the erase process andsemi-transparentization process of the information display object, thecontent of the cancel process of following by the information displayobject, the contents of the movement process and size change process ofthe information display object, or the content of the color informationchange process of the information display object according to the changeangle, the change speed, or the change acceleration of the line-of-sightdirection (the change distance, the change speed, or the changeacceleration of the point-of-view position). For example, as the changeangle, the change speed, or the change acceleration of the line-of-sightdirection (the change distance, the change speed, or the changeacceleration of the point-of-view position) is greater, the displayprocessing section 120 shortens the time before the information displayobject is erased, increases the change speed of semi-transparency,shortens the time before the following is canceled, or accelerates amovement speed in the movement process or a size change speed in thesize change process. In addition, as the change angle, the change speed,or the change acceleration of the line-of-sight direction (the changedistance, the change speed, or the change acceleration of thepoint-of-view position) is smaller, the display processing section 120lengthens the time before the information display object is erased,decreases the change speed of semi-transparency, lengthens the timebefore the following is canceled, or decelerates a movement speed in themovement process or a size change speed in the size change process.

In addition, a determination range (acceptance range of theline-of-sight direction) is set to the information display object, forexample. When the gaze position of the user in the line-of-sightdirection exceeds the determination range, the determination section 115determines that the given change condition is satisfied.

For example, the determination range is set to include the informationdisplay object as seen from the point-of-view of the user. The settingof the determination range can be performed by the determination section115, for example. The determination range may be a determination volumefor three-dimensional shape in a collision process or a range fortwo-dimensional shape. The determination section 115 performs crossingdetermination to determine whether a straight line along theline-of-sight direction of the user (a straight line of vector of theline-of-sight direction) and the determination range cross each other,for example. When determining that the straight line and thedetermination range cross each other, the determination section 115determines that the given change condition is not satisfied. On theother hand, when the straight line and the determination range do notcross each other, the determination section 115 judges that the gazeposition of the user in the line-of-sight direction has exceeded thedetermination range and determines that the given change condition issatisfied. The crossing determination between the straight line alongthe line-of-sight direction and the determination range can beimplemented by a publicly known collision process (hit determination).

In this case, the determination range is desirably different betweeninformation display objects, for example. For example, as thedetermination range, a first determination range is set to a firstinformation display object out of the plurality of information displayobjects. On the other hand, as the determination range, a seconddetermination range different from the first determination range is setto a second information display object out of the plurality ofinformation display objects. For example, the first determination rangeand the second determination range are different in size as seen fromthe point-of-view of the user. For example, the smaller determinationrange (the first determination range) is set to an information displayobject positioned in a central part as seen from the point-of-view ofthe user. On the other hand, for example, the larger determination range(the second determination range larger than the first determinationrange) is set to an information display object positioned in aperipheral part as seen from the point-of-view of the user.

When a plurality of information display objects are arranged in thevirtual space, the display processing section 120 may perform the changeprocess of the display mode on the information display object positionedin the central part as seen from the point-of-view of the user. Forexample, the display processing section 120 performs the change processof the display mode on the information display object positioned in thecentral part as seen from the point-of-view of the user, but does notperform the change process of the display mode on the informationdisplay object positioned in the peripheral part as seen from thepoint-of-view of the user. The central part here refers to a region thatcrosses a straight line extended along the line-of-sight direction ofthe user, for example, which is a region of a predetermined size definedby a predetermined boundary. On the other hand, the peripheral part is aregion around the central part, which does not cross the straight line.The central part and the peripheral part are separated by thepredetermined boundary described above. As an example, the displayprocessing section 120 may perform the change process of the displaymode on the information display object crossing the straight lineextended along the line-of-sight direction of the user, but may notperform the change process of the display mode on the informationdisplay object not crossing the straight line.

When a plurality of information display objects are arranged in thevirtual space, the display processing section 120 may differentiate thecontent of the given change condition or the content of the changeprocess of the display mode between the first information display objectand the second information display object. For example, the displayprocessing section 120 differentiates the content of the changecondition or the content of the change process of the display mode foreach one of the plurality of information display objects. The displayprocessing section 120 may differentiate both the content of the changecondition and the content of the change process of the display mode. Forexample, the display processing section 120 can differentiate thecontent of the change condition by differentiating a threshold fordetermination on the change condition based on the change angle, changespeed, or change acceleration of the line-of-sight direction (the changedistance, change speed, or change acceleration of the point-of-viewposition). Alternatively, the display processing section 120 candifferentiate the content of the change condition by differentiating thesize of the determination range described above. Otherwise, the displayprocessing section 120 can differentiate the content of the changeprocess of the display mode of the information display object bydifferentiating the contents of the erase process andsemi-transparentization process of the information display object, thecontent of the cancel process of following by the information displayobject, the contents of the movement process and size change process ofthe information display object, or the content of the color informationchange process of the information display object.

The display processing section 120 may perform the change process of thedisplay mode according to a depth value of the information displayobject. The depth value indicates a distance of the information displayobject in a depth direction as seen from the point-of-view of the user,which is a value of a Z coordinate axis in a point-of-view coordinatesystem (camera coordinate system), for example. For example, the depthvalue of the first information display object is a first depth value,and the depth value of the second information display object is a seconddepth value different from the first depth value. In this case, thedisplay processing section 120 performs the change process of thedisplay mode on the first information display object (for example, theinformation display object on the front side as seen from thepoint-of-view), but does not perform the change process of the displaymode on the second information display object (for example, theinformation display object on the back side as seen from thepoint-of-view). Otherwise, the display processing section 120 maydifferentiate the content of the change process of the display modebetween the first information display object and the second informationdisplay object.

The display processing section 120 may perform the change process of thedisplay mode according to the length of a time elapsed since the givenchange condition is satisfied. For example, with the length of a timeelapsed since the change condition is satisfied as a parameter, thedisplay processing section 120 performs the erase process andsemi-transparentization process of the information display object, thecancel process of following by the information display object, themovement process and size change process of the information displayobject, or the color information change process of the informationdisplay object. For example, the display processing section 120 changesthe degree of semi-transparency in the semi-transparentization processbased on the length of a time elapsed since the change condition issatisfied. For example, the display processing section 120 sets thedegree of semi-transparency such that the information display objectbecomes closer to transparency with increase in the length of a timeelapsed. Alternatively, the display processing section 120 may set themovement speed in the movement process or the size change speed in thesize change process based on the length of a time elapsed since thechange condition is satisfied.

After the change process of the display mode of the information displayobject, when the determination section 115 determines that a givenreturn condition is satisfied, the display processing section 120performs a return process of the change process of the display mode ofthe information display object. The return process is a process ofreturning to the state before the change of the display mode orapproaching the state before the change of the display mode. Forexample, when the erase process of the information display object hasbeen performed by the change process of the display mode, the displayprocessing section 120 performs a return process of displaying again theerased information display object. When the semi-transparentizationprocess of the information display object has been performed by thechange process of the display mode, the display processing section 120performs a return process of returning the semi-transparent informationdisplay object to the non-semi-transparent information display object.When the cancel process of following by the information display objecthas been performed by the change process of the display mode, thedisplay processing section 120 disables the cancel process and performsa return process of causing the information display object to follow theline-of-sight direction. When the movement process or the size changeprocess of the information display object has been performed by thechange process of the display mode, the display processing section 120performs a return process of returning the information display object tothe state before the movement or the state before the size change. Whenthe color information change process of the information display objecthas been performed by the change process of the display mode, thedisplay processing section 120 performs a return process of returningthe color information of the information display object to the colorinformation before the change.

In this case, when a given time has elapsed after the change process ofthe display mode, or when no change in the line-of-sight direction ofthe user has been detected for a given time after the change process ofthe display mode, or when the gaze of the user on a given object hasbeen detected after the change process of the display mode, thedetermination section 115 determines that the given return condition issatisfied. Then, the display processing section 120 executes the returnprocess of the change process of the display mode of the informationdisplay object. For example, after the change process of the displaymode, the determination section 115 measures the length of a timeelapsed. When it is determined that a predetermined given time haselapsed, the display processing section 120 executes the return process.After the change process of the display mode, the display processingsection 120 performs a detection process of a change in theline-of-sight direction of the user. When determining that the change inthe line-of-sight direction is as small as equal to or smaller than apredetermined change amount, the display processing section 120 executesthe return process. After the change process of the display mode, whenit is detected that the user has gazed at a given object for a givenperiod of time, the display processing section 120 executes the returnprocess.

The display processing section 120 may differentiate brightness of thedisplay image on the HMD 200 between when the change process of thedisplay mode of the information display object is not performed and whenthe change process of the display mode of the information display objectis performed. The brightness of the display image refers to thebrightness of the entire display image, which can be prescribed by theaverage luminance (lightness) of all the pixels constituting the displayimage, for example. When the change process of the display mode of theinformation display object is not performed but the information displayobject is on normal display, the display processing section 120 darkensthe display image on the HMD 200, for example. On the other hand, whenthe change process of the display mode of the information display objectis performed and the information display object is not displayed or issemi-transparent, the display processing section 120 brightens thedisplay image on the HMD 200, for example.

The virtual camera control section 118 controls the virtual camera tofollow a change in the point-of-view of the user based on trackinginformation of the point-of-view information of the user.

For example, the input processing section 102 (input reception section)acquires tracking information of point-of-view information of the userwearing the HMD 200. For example, the input processing section 102acquires tracking information (point-of-view tracking information) forpoint-of-view information that is at least one of the point-of-viewposition and the line-of-sight direction of the user. For example, thetracking information can be acquired by performing a tracking processfor the HMD 200. The point-of-view position and the line-of-sightdirection of the user may be directly acquired by the tracking process.For example, the tracking information may include at least one of changeinformation (a value of change in the coordinates of the point-of-viewposition) about the point-of-view position from the initialpoint-of-view position of the user and change information (a value ofchange in the rotational angle about the rotation axis of theline-of-sight direction) of the line-of-sight direction from the initialline-of-sight direction of the user. Based on the change informationabout the point-of-view information included in such trackinginformation, the point-of-view position and/or the line-of-sightdirection of the user (the information about the position and theorientation of the head of the user) can be identified.

The virtual camera control section 118 changes the point-of-viewposition and/or the line-of-sight direction of the virtual camera basedon the acquired tracking information (information about at least one ofthe point-of-view position and the line-of-sight direction of the user).For example, the virtual camera control section 118 sets the virtualcamera so that the point-of-view position/line-of-sight direction(position/orientation) of the virtual camera in the virtual spacechanges in accordance with the change in the point-of-viewposition/line-of-sight direction of the user in the real space. Thus,the virtual camera can be controlled to follow the change in thepoint-of-view of the user, based on the tracking information about thepoint-of-view information of the user.

In the present embodiment, a virtual reality simulation process isperformed as a game process for the game played by the user. The virtualreality simulation process is a simulation process simulating a realspace event in the virtual space, and is for enabling the user tovirtually experience the event. For example, a process of moving themoving body such as a virtual user corresponding to the user in the realspace or its ridden moving body in the virtual space is performed forenabling the user to experience a change in an environment orsurroundings due to the movement.

The simulation system includes the game processing section 112 thatperforms game processes. The game processing section 112 performs aprocess of activating an effect of a command corresponding to theinformation on a character as a target of gameplay by the user and thepoint-of-view information of the user.

In the present embodiment, for example, the game processing section 112performs a process of accepting a command selected by the user from aplurality of commands. For example, the storage section 170 (commandinformation storage section) stores information on N commands that canbe selected by the user. For example, M (M<N) commands are selectedautomatically or by the user from the N commands, and a command setcalled deck is organized. During gameplay, the user selects a desiredcommand from the M commands. Accordingly, the selected command isaccepted as a command of activation target.

The game processing section 112 performs the process of activating theeffect of the accepted command as a game process. For example, theeffect of each command is associated with the command and is stored ascommand information in the storage section 170. The game processingsection 112 performs the process of activating the effect associatedwith the command selected by the user.

Based on the result of activation of the effect of the command, the gameprocessing section 112 performs a game process such as a process ofcalculating game parameters, a process of proceeding the game, a processof calculating game performance (game result), or a process ofcontrolling an icon corresponding to the command.

Specifically, the game processing section 112 performs the calculationprocess of increasing or decreasing the values of the game parameters.Based on the game parameters after the calculation process, the gameprocessing section 112 performs various game processes such ascontrolling the action (behavior) of a character, branching the story,and generating a game event. For example, the game processing section112 performs a game process such that the story is made progress by theuser and the character as a target of gameplay by the user.

Based on the result of the game process described above, the displayprocessing section 120 performs a display process of game images. Forexample, the display processing section 120 performs the process ofdisplaying game images on the display section 220 of the HMD 200according to the results of the calculation of the game parameters, theresults of the game progress process, the results of the gameperformance calculation, and the results of the icon control process. Inaddition, the sound processing section 130 performs a process ofoutputting BGM, sound effects, or game sounds such as voices from thesound output section 192 based on the results of the game process.

In the present embodiment, the game processing section 112 performs aprocess of activating the effect of the command according to theinformation on the character as a target of gameplay by the user and thepoint-of-view information of the user. For example, the game processingsection 112 performs a process of activating the effect of the commandaccording to the information set (specified) by the characterinformation and the point-of-view information of the user (for example,the positional relationship information, the line-of-sight relationshipinformation, or the gaze information).

Specifically, the game processing section 112 performs a process ofchanging at least one of the degree of the effect of the command and thecontents of the effect of the command according to the characterinformation and the point-of-view information of the user. For example,the game processing section 112 performs the activation process of thecommand such that, when the information set (specified) by the characterinformation and the point-of-view information of the user (for example,the positional relationship information, the line-of-sight relationshipinformation, or the gaze information) changes, at least one of thedegree of the effect of the command and the contents of the effect ofthe command changes accordingly.

In this case, the character information is at least one of positioninformation, direction information, game parameter information, bodypart information, posture information, and type information of thecharacter, for example. The character is a gameplay element appearing inthe game, which represents a person, an animal, a robot, or a movingbody in the real world, for example, and is displayed as an object inthe game image.

Specifically, the character as a target of gameplay by the user is acharacter that is a partner of gameplay by the user, for example. Takinga communication game with an opposite-sex character (personalrelationship simulation game) as an example, the character is a partner(candidate) of communication with the user. For example, in the presentembodiment, the display processing section 120 generates a first personpoint-of-view game image for the user, and displays the generated firstperson point-of-view image on the display section 220 of the HMD 200.This provides the user with virtual reality in which the user feels asif he or she enters into a game space (CG animation world). Thecharacter appears as a target of gameplay by the user in the game space(the object space or the virtual three-dimensional space). Accordingly,the user can feel the virtual reality as if a real person (thecharacter) exists in front of him or her. As the character as a targetof gameplay by the user, a character appearing as the user's foe orfriend in a war game, RPG game, or sports game may be used.

The game parameters are parameters that are used in the game processes(the game progress process and others). For example, the game parametersare parameters representing the status or performance of the characterand the user. For example, the game parameters of the character may bevarious parameters such as status parameter representing the mind orstatus of the character, a parameter representing the character'sperformance, a parameter related to the character's action, or aparameter related to the character's belongings. The game parameters ofthe character may be parameters representing the character's evaluationof the user.

The body part information of the character includes information on thekinds of the body parts (body part objects) of the character (such ashead, chest, hip, legs, or hands), information on the positions of thebody parts (relative positions to the representative position of thecharacter), or information on the shapes of the body parts (the shapesof the body part objects). The posture information of the character isinformation on the posture specifying the motion of the character suchas sitting, standing, walking, or running, for example. For example,when the character is represented by a skeleton model, shape informationof the skeleton corresponds to the posture information of the character.The kind information of the character is called the type or attribute ofthe character. For the opposite-sex character, the character type may bean active type, a serious type, a childish type, or a neat type.

The point-of-view information of the user is at least one of thepoint-of-view position information and the line-of-sight directioninformation of the user, and the attribute information of the virtualcamera, for example. The point-of-view position information isinformation on the position set as the point-of-view of the user in thegame space (the object space or the virtual three-dimensional space),for example. The line-of-sight direction information is informationindicating the line-of-sight direction of the user at thatpoint-of-view. The virtual camera attribute information is informationon the virtual camera set at the point-of-view of the user and may bevarious kinds of information such as angle-of-view information andbinocular disparity information in stereoscopic viewing, for example.

Further specifically, the game processing section 112 performs theprocess of activating the effect of the command according to thepositional relationship information between the user and the character.For example, in the case of first person point-of-view, the gameprocessing section 112 activates the effect of the command according tothe positional relationship information between the point-of-viewposition of the user and the position (representative position) of thecharacter. The positional relationship information is informationindicating a relationship between the position of the user (thepoint-of-view position or the position of the user character) and theposition of the character (the representative position or thepoint-of-view position). The positional relationship information may bethe distance between the user and the character (the distance in thegame space), for example, but is not limited to this. For example, thepositional relationship information may be information on directionalrelationship between the user and the character.

The game processing section 112 performs a process of activating theeffect of a command corresponding to the line-of-sight relationshipinformation indicating the relationship between the user's line-of-sightand the character. The line-of-sight relationship information isinformation indicating a relationship between the user's line-of-sightand the character, and for example, information indicating correlationbetween the user's line-of-sight and the character. For example, theline-of-sight relationship information is information relating to thedirection in which the user's line-of-sight is oriented with respect tothe position (representative position) and the body parts of thecharacter.

The game processing section 112 also performs a process of activatingthe effect of a command according to the gaze information of the user.The gaze information is at least one of gazed part information asinformation on the body part of the character gazed at by the user,presence-or-absence of gaze information as information whether the usergazes at the character, gaze time information indicating the time duringwhich the user gazes at the character, and gaze distance informationindicating the distance between the user and the character when the useris gazing at the character.

2. Tracking Process

Next, an example of a tracking process will be described. FIG. 2Aillustrates an example of the HMD 200 used in the simulation system inthe present embodiment. The HMD 200 illustrated in FIG. 2A is providedwith a plurality of light emitting elements 231 to 236. For example,these light emitting elements 231 to 236 are implemented with LEDs orthe like. The light emitting elements 231 to 234 are provided on thefront surface side of the HMD 200, and the light emitting element 235and the light emitting element 236 (not illustrated) are provided on theback surface side. These light emitting elements 231 to 236 emit lightin a wavelength band of visible light (light emission) for example.Specifically, the light emitting elements 231 to 236 emit light ofcolors different from each other. The image capturing section 150illustrated in FIG. 2B is provided on the forward side of a user PL tocapture an image of the light from the light emitting elements 231 to236. Thus, the captured image obtained by the image capturing section150 includes spots of the light from the light emitting elements 231 to236. The head (HMD) of the user PL is tracked with image processingexecuted on the captured image. Thus, the three-dimensional position andthe facing direction of the head of the user PL (the point-of-viewposition and the line-of-sight direction) are detected.

For example, as illustrated in FIG. 2B, the image capturing section 150is provided with first and second cameras 151 and 152. The position ofthe head of the user PL in a depth direction and the like can bedetected by using first and second captured images respectively capturedby the first and the second cameras 151 and 152. Based on motiondetection information obtained by the motion sensor provided to thesensor section 210 of the HMD 200, the rotational angle (line-of-sight)of the head of the user PL can also be detected. Thus, with such an HMD200, the display section 220 of the HMD 200 can display an image (animage as viewed from the virtual camera corresponding to thepoint-of-view of the user) of the virtual space (virtualthree-dimensional space) corresponding to the direction in which theuser PL is facing regardless of which of all 360 degrees directions heor she is facing. LEDs emitting infrared light, instead of the visiblelight, may be used as the light emitting elements 231 to 236.Furthermore, another method such as one using a depth camera or the likemay be employed to detect the position and/or movement of the head ofthe user and the like.

The HMD 200 is provided with a headband 260 and a headphone terminal notillustrated. The user PL can hear game sound having undergone athree-dimensional acoustic process, for example, with a headphone 270(sound output section 192) connected to the headphone terminal.

The user PL holds game controllers 166 and 167 as illustrated in FIG.2A. The game controllers 166 and 167 corresponds to the operationsection 160 illustrated in FIG. 1. The game controllers 166 and 167 areprovided with light emitting elements 237 and 238. The positions andothers of the game controllers 166 and 167 are detected in the samemanner as in the case of the HMD 200 by imaging light from the lightemitting elements 237 and 238 by the image capturing section 150. Thus,the user PL can input various kinds of operation information. The usermay be enabled to input operation information through a head nodding orshaking action to be detected by the tracking process.

FIG. 3A illustrates another example of the HMD 200 used in thesimulation system in the present embodiment. As illustrated in FIG. 3A,the HMD 200 is provided with a plurality of light receiving elements(photodiodes) 201, 202, and 203. The light receiving elements 201 and202 are provided on the front surface side of the HMD 200, whereas thelight receiving element 203 is provided on the right side surface of theHMD 200. Note that unillustrated light receiving elements are furtherprovided on the left side surface, the upper surface, or the like of theHMD. The HMD 200 is provided with the headband 260, and the user PLwears the headphone 270.

As illustrated in FIG. 3B, base stations 280 and 284 are installed inthe periphery of the user PL. The base station 280 is provided withlight emitting elements 281 and 282, and the base station 284 isprovided with light emitting elements 285 and 286. For example, thelight emitting elements 281, 282, 285, and 286 are implemented with LEDsthat emit laser beams (such as infrared laser beams). The base stations280 and 284 radially emit laser beams by using the light emittingelements 281, 282, 285, and 286, for example. With the light receivingelements 201 to 203 provided to the HMD 200 in FIG. 3A and the likereceiving the laser beams from the base stations 280 and 284, thetracking of the HMD 200 is implemented so that the position and thefacing direction of the head of the user PL (point-of-view position,line-of-sight direction) can be detected.

As illustrated in FIG. 3A, the game controllers 166 and 167 held by theuser PL are provided with light receiving element 204 to 206 and 207 to209. The positions and others of the game controllers 166 and 167 aredetected in the same method as in the case of the HMD 200 by receivinglaser beams from the light emitting elements 281, 282, 285, and 286 ofthe base stations 280 and 284 by the light receiving elements 204 to 206and 207 to 209. Thus, the user PL can input various kinds of operationinformation.

The tracking process for detecting the point-of-view position and/orline-of-sight direction (the position and/or direction of the user) ofthe user is not limited to the method described with reference to FIG.2A to FIG. 3B. For example, the tracking process may be implementedsolely by the HMD 200 by using the motion sensor and the like providedto the HMD 200. Specifically, the tracking process is implementedwithout providing external devices such as the image capturing section150 in FIG. 2B and the base stations 280 and 284 in FIG. 3B.Alternatively, various point-of-view tracking processes such as knowneye tracking, face tracking, or head tracking can be employed to detectthe point-of-view information such as the point-of-view position and theline-of-sight direction of the user. In an example of employing the eyetracking, the position, the shape, and the like of the pupil of each ofthe left and the right eyes of the user are recognized. Then, thetracking process is implemented with the point-of-view information onthe user acquired by identifying the positions and the line-of-sightdirections of the left and the right eyes. For example, the eye trackingcan be implemented by capturing images of the left and the right eyes ofthe user with the image capturing section, and performing an imagerecognition process, for the pupil or the like, on the captured images.When the face tracking is employed, an image of the face of the user iscaptured by the image capturing section, and an image recognitionprocess for the face is performed. Then, the tracking process isimplemented with the point-of-view position and the line-of-sightdirection of the user obtained by identifying the position and theorientation of the face of the user based on a result of the imagerecognition process.

3. Method According to the Present Embodiment

Next, a method according to the present embodiment is described indetail.

3.1 Description on Game

First, an overview of the game implemented with the method according tothe present embodiment is described. This game is a communication game(personal relationship simulation game) in which the user communicateswith an opposite-sex character to teach subjects, offer various kinds ofadvice, answer questions, and clear various events, thereby creating afavorable image of the user in the character. As an example of a gameemploying the method according to the present embodiment, acommunication game with an opposite-sex character or the like will bemainly described below. However, the game employing the method accordingto the present embodiment is not limited to this but the methodaccording to the present embodiment is also applicable to various gamessuch as a battle game, an RPG game, a sports game, and an action game.

FIG. 4 to FIG. 6 illustrate examples of game images generated by thesimulation system (game system) according to the present embodiment. Inthe game implemented according to the present embodiment, the user(player) visits the home of a character CH as a home tutor to teachsubjects and the like. The game has a plurality of stories in each ofwhich the character CH of a different type appears. The user attains hisor her purpose in each of the stories while enjoying intimatecommunication with the character CH.

The user wears the HMD 200 to play the game. When the user directshis/her line-of-sight to each of all the directions at 360 degrees, theimage to be seen in the direction is displayed on the display section220 of the HMD 200. For example, referring to FIG. 4, when the userdirects his/her line-of-sight to the character CH, the image of thecharacter CH being studying is displayed on the display section 220. Onthe other hand, when the user turns his/her head and directs his/herline-of-sight to the rear side of the user, the image of the furnitureand others behind the character CH in the room is displayed on thedisplay section 220. Therefore, the user can feel virtual reality as ifhe/she has entered in the game world. That is, it is possible to offervirtual reality to the user as if he/she has actually entered the roomand is teaching subjects to the opposite-sex person in front of him/herso that the user can feel an increased sense of immersion in the game.

In a game image illustrated in FIG. 4, objects such as the character CH(a moving body in a broad sense), a book held by the character CH (anitem in a broad sense), a curtain CT (a fixed object in a broad sense),and a wall WL (a background in a broad sense) are seen. In addition,information display objects DSA and DSB are displayed.

The information display object DSA is a display object for descriptiondisplay. For example, the information display object DSA describes thatthe character CH is reading the book. The information display object DSBis a display object for status display. As described later withreference to FIG. 7, the information display object DSB provides variouskinds of status information of the character CH.

Referring to FIG. 4, when the user approaches the character CH and turnshis/her line-of-sight to the character CH, a game image illustrated inFIG. 5 is displayed. FIG. 5 further illustrates information displayobjects DSC, DSD1, and DSD2. The information display object DSC is adisplay object for conversation display. For example, the informationdisplay object DSC shows the character CH's utterance “What's up?” Theinformation display objects DSD1 and DSD2 are display objects forcommand selection (command input). For example, each of the displayobjects shows a command for the answer (YES or NO) to the utterance ofthe character CH.

Referring to FIG. 5, when the user selects the command for “YES”, a gameimage as illustrated in FIG. 6 is displayed. FIG. 6 illustratesinformation display objects DSE1, DSE2, and DSE3. The informationdisplay objects DSE1, DSE2, and DSE3 are display objects (icons) forcommand selection. For example, the information display objects DSE1,DSE2, and DSE3 are associated with commands CM1, CM2, and CM3. Forexample, when the user selects the information display object DSE1, theeffect of the command CM1 “concentrate more” is activated. When the userselects the information display object DSE2, the effect of the commandCM2 “scold” is activated. When the user selects the information displayobject DSE3, the effect of the command CM3 “speak” is activated.

The selection of the information display object (DSE1, DSE2, DSE3, DSD1,or DSD1) corresponding to the command is made by using the operationsection 160 illustrated in FIG. 1. For example, the user uses theoperation section 160 to perform a command operation of selecting theinformation display object (icon) corresponding to the desired commandto activate the effect of the command corresponding to the selectedinformation display object. Referring to FIG. 2A and FIG. 3A, forexample, the user performs such a command operation by maneuvering thegame controllers 166 and 167 by the both hands. Alternatively, the usermay perform the command operation by using a game controller (notillustrated) connected in a wired manner. Alternatively, the user withthe HMD 200 on his/her head may perform the command operation by turninghis/her line-of-sight direction to the information display objectcorresponding to the command or making an action such as shaking his/herhead vertically or horizontally.

Referring to FIG. 6, the user selects the information display objectDSE3 corresponding to the command CM3 “speak”. Accordingly, the effectof the command CM3 “speak” is activated. For example, the command CM3 isassociated with voice data indicating the words corresponding to “speak”(for example, “Yoo-hoo!”, “Are you working hard?” or “Are you studyinghard?”). When the command CM3 is activated, the voice data is reproducedand the voice of the words (sentence) is output from the sound outputsection 192 illustrated in FIG. 1.

In addition, when the command is activated (the effect of the command isactivated), the game parameters of the character CH are also influenced.For example, when the command CM3 “speak” is activated, thecorresponding game parameters of the character CH also change.

For example, referring to FIG. 4 to FIG. 6, the information displayobject DSB is displayed to inform the user about the status of the gameparameters indicating various states of the character CH. Theinformation display object DSB informs the user about the current mentalstate, motivation, concentration, pulse, and others of the character CHas illustrated in FIG. 7, for example. The mental state, motivation,concentration, and pulse are prepared as game parameters of thecharacter CH.

The mental state of the character CH is represented by color, forexample. In the game of the present embodiment, for example, assignmentsin a lesson are given and the lesson is ended after a lapse of apredetermined time. In addition, according to the achievement amounts ofthe assignments at the end of the lesson, for example, the user'sperformance of the game is determined. For example, when the user'sperformance of the game is high, the user is given a card of a command(action) that is highly effective or highly rare.

The assignments in the lesson are given red, green, and blue attributes,for example, and the mental state of the character CH also has red,green, and blue components. For example, when the mental state of thecharacter CH has a large amount of red component, the assignment withthe red attribute can be cleared up immediately, whereas the assignmentswith the green and blue attributes are cleared up at a slow pace.Similarly, when the mental state of the character CH has a large amountof green component, the assignment with the green attribute can becleared up immediately, whereas the assignments with the red and blueattributes are cleared up at a slow pace. Therefore, the user checks themental state of the character CH displayed in the information displayobject DSB and causes the character CH to clear up the assignment withthe attribute suited to the mental state so that the amount of theassignment achieved increases and the lesson can be properly completed.

In addition, as illustrated in FIG. 7, the game parameters of motivationand concentration are set to the character CH. The values of the gameparameters of motivation and concentration vary between 0 to 100(%), forexample, and the motivation and concentration become higher as thevalues are larger. When the value of the motivation becomes high, thespeed of clearing up the assignment increases. On the other hand, thevalue of the concentration becomes lower with a lapse of time, and theconcentration becomes larger at an increased rate or becomes unlikely tobe lower depending on the mental state of the character CH. In addition,when the value of the concentration becomes lower than a predeterminedvalue and the concentration of the character CH becomes lost, thecharacter CH needs to take a break.

The pulse shown on the information display object DSB illustrated inFIG. 7 functions as a game parameter representing the heart-poundingfeeling and favorable impression the character CH has on the user, forexample. For example, when the degree of heart-pounding feeling of thecharacter CH becomes higher, the pulse also becomes higher, themotivation improves, the concentration is less likely to be lower, andthe speed of clearing up the assignment increases.

3.2 Change Process of Display Mode

As illustrated in FIG. 4 to FIG. 6, in the game implemented by thepresent embodiment, various information display objects are displayed.In a conventional image generation system that displays pseudothree-dimensional images on a display section without using an HMD,two-dimensional display objects (sprite and the like) aretwo-dimensionally displayed as information display objects on aprojection screen.

However, in a virtual reality (VR) system using an HMD, whentwo-dimensional display objects are displayed as information displayobjects for description display, status display, command selection, menudisplay, and others, the virtual reality for the user may becomedeteriorated or the screen may become hard to see. For example, in a VRspace in which three-dimensional objects such as the character CH, thebook BK, the curtain CT, and the wall WL illustrated in FIG. 4 aredisplayed in a stereoscopic manner, when only the information displayobjects are two-dimensionally displayed on the projection screen, thevirtual reality becomes deteriorated. In addition, when a large numberof two-dimensional information display objects are displayed, theseinformation display objects overlap two-dimensionally or some objectsare hidden behind the information display objects so that the screenbecomes very hard to see.

Accordingly, in the present embodiment, at least one information displayobject is arranged in the line-of-sight direction of the user (the fieldof view range or the front direction of the user). For example, theinformation display object is formed from an object havingthree-dimensional coordinate values and is arranged in the virtual spacesimilarly to the other objects in the virtual space. Specifically, theinformation display object controlled to follow a change in thepoint-of-view of the user (a change in at least one of the line-of-sightdirection and the point-of-view position) is disposed in the virtualspace. Further specifically, the information display object is disposedat a given depth position as seen in the line-of-sight direction of theuser to follow a change in the point-of-view of the user.

FIG. 8A to FIG. 8D are diagrams describing a display method ofinformation display objects. Referring to FIG. 8A, for example, VPrepresents the point-of-view of the user PL wearing the HMD, and VLrepresents the line-of-sight direction of the user PL. The point-of-viewVP and the line-of-sight direction VL are equivalent to thepoint-of-view and the line-of-sight direction of the virtual cameracorresponding to the user PL.

Referring to FIG. 8A, information display objects DS1 and DS2 aredisposed respectively at positions of depth values Z1 and Z2 as seen inthe line-of-sight direction VL of the user PL. The depth values Z1 andZ2 are coordinate values of a Z coordinate axis in a point-of-viewcoordinate system (camera coordinate system), for example. Theinformation display objects DS1 and DS2 are implemented by billboardpolygons, for example. Specifically, the information display objects DS1and DS2 are implemented by mapping images of characters, icons, orgraphics for description display, status display, conversation display,command selection display, or menu display as described above withreference to FIG. 4 to FIG. 6 to the billboard polygons. The billboardpolygons (billboard primitive surfaces) are polygons (primitivesurfaces) that are arranged to face (appropriately face) thepoint-of-view VP of the user PL (virtual camera). For example, thebillboard polygons as the information display objects DS1 and DS2 arearranged to be orthogonal (approximately orthogonal) to theline-of-sight direction VL (line-of-sight direction vector). Forexample, the information display objects DS1 and DS2 are arranged suchthat normal directions of surfaces of the billboard polygons areparallel to the line-of-sight direction VL. The arrangement direction ofthe billboard polygons is not limited to the direction as describedabove but the billboard polygons can be set in any of variousarrangement directions. In addition, the information display objects maybe implemented by display objects different in form from billboardpolygons. For example, the information display objects may beimplemented by three-dimensional objects formed from a plurality ofprimitive surfaces (polygons).

In the present embodiment, the information display objects DS1 and DS2are controlled to follow a change in the line-of-sight direction VL ofthe user PL. Referring to FIG. 8B, for example, the line-of-sightdirection VL of the user PL changes to the right side. For example, theuser PL wearing the HMD faces to the right, and thus the line-of-sightdirection of the virtual camera in the virtual space also changes to theright side. In this case, as illustrated in FIG. 8B, the positions anddirections of the information display objects DS1 and DS2 change tofollow the change in the line-of-sight direction VL. Referring to FIG.8B, for example, the billboard polygons constituting the informationdisplay objects DS1 and DS2 are also arranged to face the point-of-viewVP of the user PL.

Referring to FIG. 8C, the line-of-sight direction VL of the user PLchanges to the left side. For example, the user PL wearing the HMD facesto the left, and thus the line-of-sight direction of the virtual camerain the virtual space also changes to the left side. In this case, asillustrated in FIG. 8C, the positions and directions of the informationdisplay objects DS1 and DS2 change to follow the change in theline-of-sight direction VL. Referring to FIG. 8C, for example, thebillboard polygons constituting the information display objects DS1 andDS2 are also arranged to face the point-of-view VP of the user PL.

Referring to FIG. 8D, the line-of-sight direction VL of the user PLchanges upward. In this case, as illustrated in FIG. 8D, the positionsand directions of the information display objects DS1 and DS2 change tofollow the change in the line-of-sight direction VL. Similarly, when theline-of-sight direction of the user PL changes downward, the positionsand directions of the information display objects DS1 and DS2 change tofollow the change in the line-of-sight direction.

In this manner, according to the display method illustrated in FIG. 8Ato FIG. 8D, the information display objects are three-dimensionallyarranged in the virtual space similarly to the other objects in thevirtual space. Therefore, it is possible to solve the problem ofdeterioration in the virtual reality for the user due to the display ofthe information display objects. Specifically, according to aconventional method by which two-dimensional information display objectsare planarly displayed, there arises a problem of deterioration in thevirtual reality due to the presence of the information display objects.However, according to the method in the present embodiment by whichinformation display objects are three-dimensionally arranged in thevirtual space, this problem can be prevented. In addition, arrangingthree-dimensionally the information display objects makes it possibleto, even when a plurality of information display objects are to bedisplayed, display the information display objects in an appropriaterepresentation that is easy to see for the user. For example, asillustrated in FIG. 8A to FIG. 8D, it is possible to arrange a pluralityof information display objects (DS1 and DS2) at different depth values(Z1 and Z2) so that the information display objects can be displayed inan appropriate representation that is easy to see for the user. Inaddition, according to the display method illustrated in FIG. 8A to FIG.8D, the information display objects are always displayed in front of theuser. Therefore, even when the line-of-sight direction of the userchanges to any of various directions, the information display objectsare displayed in front of the user so that various kinds of informationnecessary for the user can be appropriately presented using theinformation display objects.

The display method of the information display objects is not limited tothe method illustrated in FIG. 8A to FIG. 8D but various modificationexamples are possible. For example, the information display object isarranged at least in the line-of-sight direction of the user and may notfollow a change in the point-of-view. For example, a specificinformation display object may not follow a change in the point-of-viewof the user. For example, the information display object for displayingthe status of a specific object (for example, a character as a target ofgameplay by the user) may be controlled to be displayed in the vicinityof the specific object.

FIG. 8A to FIG. 8D illustrate the case in which the information displayobjects follow a change in the line-of-sight direction of the user.Alternatively, each of the information display objects may follow achange in the point-of-view position of the user. For example, when thepoint-of-view position of the user (user position) moves upward,downward, leftward, or rightward, the information display object ismoved upward, downward, leftward, or rightward to follow this movement.The information display object may follow only a change in theline-of-sight direction of the user, may follow only a change in thepoint-of-view position, or may follow both a change in the line-of-sightdirection and a change in the point-of-view position. However, thefollowing description will be given mainly as to the case in which achange in the point-of-view of the user is a change in the line-of-sightdirection of the user as an example, for the sake of simplicity.

As illustrated in FIG. 8A to FIG. 8D, in the present embodiment, amethod for arranging and displaying an information display object in theline-of-sight direction of the user is employed. Accordingly, theinformation display object is arranged at any time in the frontdirection of the user, which makes it possible to perform appropriateinformation transmission to the user by the information display object.In addition, instead of displaying the information display objecttwo-dimensionally, arranging the information display object in the samemanner as other objects such as a character in the virtual spaceachieves appropriate display of the information display object withoutdeterioration in the virtual reality for the user.

However, according to the method by which to arrange the informationdisplay object in the line-of-sight direction of the user, when theline-of-sight direction of the user changes greatly (the point-of-viewposition changes greatly), an inappropriate display image may bedisplayed on the HMD 200. In the system using the HMD, the VR spacespreads over the entire periphery of the field of view of the user.Accordingly, the user may move the line-of-sight direction to lookaround the VR space. In this case, if the information display object isdisplayed in the display mode in which the line-of-sight direction isoriented to the front direction, the display may be an inappropriate forthe user.

Accordingly, in the present embodiment, the process of changing thedisplay mode of the information display object is performed when a givenchange condition is satisfied by a great change in the line-of-sightdirection or the like. In addition, when the user moves theline-of-sight direction not only to the front direction but also in sucha manner as to look around, the process of changing the display mode ofthe information display object is performed such that the appropriateimage is displayed on the HMD.

For example, referring to FIG. 9A and FIG. 9B, the display mode of theinformation display object is changed. That is, as illustrated in FIG.9A, the information display object DS is arranged in the line-of-sightdirection VL of the user PL. Specifically, as described above withreference to FIG. 8A to FIG. 8D, the information display object DS isarranged to follow a change in the line-of-sight direction VL of theuser PL.

Then, the user PL changes the line-of-sight direction VL as illustratedin FIG. 9B. For example, in the system using the HMD, the VR spacespreads over the entire periphery. Accordingly, the user PL may changegreatly the line-of-sight direction VL as illustrated in FIG. 9B to lookaround the VR space. In this case, in the present embodiment, it isdetermined whether a change condition described later is satisfied bythe change in the line-of-sight direction VL. When it is determined thatthe change condition is satisfied, the change process of the displaymode is performed to erase the display of the information display objectDS or semi-transparentize the information display object DS asillustrated in FIG. 9B. The erase process of the information displayobject DS can be implemented by excluding the information display objectDS from a display list (display object list), for example. Thesemi-transparentization process of the information display object DS canbe performed by bringing the semi-transparency (a value) of theinformation display object DS closer to transparency, for example.Alternatively, the information display object DS may be changed intoframe models or the like, for example, such that characters, icons, orgraphics for description display or status display are erased.

When the user PL faces the front direction as illustrated in FIG. 9A,arranging the information display object DS in the line-of-sightdirection VL of the user PL allows appropriate information transmissionusing the information display object DS. For example, as illustrated inFIG. 4 to FIG. 6, various kinds of information for description display,status display, conversation display, command selection, or menu displaycan be appropriately transmitted to the user PL.

On the other hand, if the information display object DS is displayedwhen the user PL changes the line-of-sight direction VL to look aroundthe VR space, the information display object DS blocks the user PL whocannot properly look around.

In this respect, in the present embodiment, when the line-of-sightdirection VL changes greatly (when the point-of-view position changesgreatly), the process of erasing or semi-transparentizing theinformation display object DS is performed as illustrated in FIG. 9B.Accordingly, the information display object DS does not block the fieldof view of the user so that the user PL can properly look around the VRspace. The following description will be given mainly as to the case inwhich the line-of-sight direction VL of the user PL changes upward as anexample. However, the change process of the display mode of theinformation display object can also be performed when the line-of-sightdirection VL changes greatly to other directions such as downward,leftward, and rightward directions.

Examples of the change process of the display mode of the informationdisplay object include not only the erase process andsemi-transparentization process of the information display object asillustrated in FIG. 9A and FIG. 9B but also the cancel process offollowing by the information display object to the line-of-sightdirection of the user, the movement process of the information displayobject, the size change process of the information display object, thecolor information change process of the information display object, andothers.

For example, referring to FIG. 10A and FIG. 10B, as the change processof the display mode, the cancel process of following by the informationdisplay object DS to the line-of-sight direction VL of the user PL isperformed. For example, in the present embodiment, as described abovewith reference to FIG. 8A to FIG. 8D, the information display object DSis controlled to follow a change in the line-of-sight direction VL ofthe user PL.

In contrast to this, referring to FIG. 10A and FIG. 10B, when the changecondition is satisfied by a change in the line-of-sight direction VL,the control of following is canceled. Accordingly, as illustrated inFIG. 10B, even when the line-of-sight direction VL of the user PLchanges, the position of the information display object DS does notchange. Therefore, the information display object DS hardly comes intothe field of view of the user PL turning the line-of-sight direction VLupward, which makes it possible to prevent a situation in which theinformation display object DS blocks the field of view.

Referring to FIG. 11A and FIG. 11B, as the change process of the displaymode, the movement process of the information display object DS isperformed. Specifically, when the change condition is satisfied by achange in the line-of-sight direction VL (a change in the point-of-viewin a broad sense, which is also applied to the following description),the information display object DS is moved from the central part to theperipheral part of the field of view of the user PL facing upward asillustrated in FIG. 11B. For example, since the line-of-sight directionVL of the user PL changes upward, the information display object DS ismoved in the opposite downward direction (obliquely downward direction).Accordingly, the information display object DS is moved to theperipheral part of the field of view of the user PL, which makes itpossible to prevent a situation in which the information display objectDS blocks the field of view.

Referring to FIG. 12A and FIG. 12B, as the change process of the displaymode, the size change process of the information display object DS isperformed. Specifically, when the change condition is satisfied by achange in the line-of-sight direction VL, the size of the informationdisplay object DS is reduced as illustrated in FIG. 12B. Reducing thesize of the information display object DS in this manner makes itpossible to decrease the area of a region in the field of view of theuser PL facing upward that is blocked by the information display objectDS. Therefore, it is possible to suppress the information display objectDS blocking the field of view in an effective manner.

In the case of performing any of the change process of the display modeas illustrated in FIG. 9A to FIG. 12B, the information display objectmay be controlled to follow a change in the line-of-sight direction witha time delay. For example, control like rubber band control in a racegame is performed. Then, after (or while) the information display objectis followed to follow with a time delay, any of the change processes ofthe display mode as illustrated in FIG. 9A to FIG. 12B is finallyperformed. For example, after the information display object is followedto follow a change in the line-of-sight direction with a time delay, theerase process or semi-transparentization process of the informationdisplay object as illustrated in FIG. 9B or the complete cancel processof following as illustrated in FIG. 10B is performed.

In addition, in the present embodiment, the change process of thedisplay mode is performed according to the type, priority, or importancelevel of the information display object. For example, when theinformation display object is of a first type (for example, a displayobject with a high importance level or priority), the display mode ofthe information display object is not changed even when theline-of-sight direction changes greatly and the change condition issatisfied. Accordingly, even when the line-of-sight direction changes,the information display object for transmitting information with a highimportance level or priority is arranged in the line-of-sight directionof the user so that the information can be properly transmitted to theuser. On the other hand, in the case where the information displayobject DS2 is of a second type (a display object with a low importancelevel or priority), when the change condition is satisfied, the displaymode of the information display object is changed. This prevents theinformation display object from blocking the field of view of the user.

For example, referring to table data illustrated in FIG. 13A,information display objects DSF1, DSF2, and DSF3 are associated withtypes TPA, TPB, and TPA. The table data is stored in the storage section170 illustrated in FIG. 1. The type TPA is a type with a low priority orimportance level, for example. Accordingly, when the line-of-sightdirection changes greatly and the change condition is satisfied, thedisplay mode of the information display object is changed. For example,the process of erasing or semi-transparentizing the information displayobject is performed. On the other hand, the type TPB is a type with ahigh priority or importance level, for example. Accordingly, even whenthe line-of-sight direction changes greatly and the change condition issatisfied, the display mode of the information display object is notchanged. For example, the erase process or semi-transparentizationprocess of the information display object is not performed.

In the table data of FIG. 13B, information display objects DSG1, DSG2,and DSG3 are associated with priorities (importance levels) PRA, PRB,and PRC. The table data is stored in the storage section 170 illustratedin FIG. 1. When the change condition is satisfied by a change in theline-of-sight direction, the change process of the display mode isperformed using priorities (importance levels) PRA, PRB, and PRCassociated with the information display objects DSG1, DSG2, and DSG3.For example, even when the change condition is satisfied, the changeprocess of the display mode is not performed on the information displayobject associated with a high priority (importance level). On the otherhand, when the change condition is satisfied, the change process of thedisplay mode is performed on the information display object associatedwith a low priority (importance level). In the case of displaying aplurality of information display objects, the information display objectas a target of the change process of the display mode is selected basedon the priority (importance level). For example, the priorities(importance levels) of a first information display object and a secondinformation display object are compared and it is judged that the firstinformation display object is higher in priority (importance level) thanthe second information display object. In this case, when the changecondition is satisfied, the change process of the display mode is notperformed on the first information display object but is performed onthe second information display object.

In this manner, in the present embodiment, it is possible to implementthe appropriate change process of the display mode according to thetype, priority, or importance level of the information display object.For example, the display mode of a specific type of information displayobject is not changed even when the line-of-sight direction changesgreatly and the change condition is satisfied. In addition, in the casewhere the information presented by the information display object ishigh in priority or importance level, the display mode is not changedeven when the change condition is satisfied. In contrast, in the casewhere the information presented by the information display object is lowin priority or importance level, the change process of the display modeof the information display object is performed on a priority basis.

In the present embodiment, the change process of the display mode isperformed using the priorities or importance levels associated with theinformation display objects as described in FIG. 13B. Accordingly,associating the priority or the importance level in advance with theinformation display object makes it possible to execute the appropriatechange process of the display mode using the priority or the importancelevel.

For example, referring to FIG. 4 to FIG. 6, the information displayobjects (DSD1, DSD2, and DSE1 to DSE3) for command selection areinformation display objects with high priorities or importance levels.For example, in the case where the line-of-sight direction of the userchanges and the change condition is satisfied, if the informationdisplay object for command selection is erased or semi-transparentized,the user cannot select (input) a command and the progress of the gamewill be stopped. Therefore, such an information display object forcommand selection is associated with a high priority or importance levelso that, even when the change condition is satisfied, the erase processor semi-transparentization process of the information display object isnot performed.

On the other hand, the information display object for descriptiondisplay (DSA) is an information display object not so high in priorityor importance level, and thus when the change condition is satisfied bya change in the line-of-sight direction of the user, for example,erasing or semi-transparentizing the information display object fordescription display would not greatly impede the progress of the game.Therefore, such an information display object for description display isassociated with a low priority or importance level, and when the changecondition is satisfied, the erase process or semi-transparentizationprocess of the information display object is performed. This makes itpossible to implement the appropriate display process reflecting thepriority or importance level of the information display object.

When both the information display object for command selection and theinformation display object for description display are displayed, forexample, the information display object as a target of the changeprocess of the display mode is selected based on the priority orimportance level. Specifically, when the change condition is satisfied,the information display object for description display is selected on apriority basis and the display mode of the selected information displayobject is changed.

In the present embodiment, the change process of the display mode may beperformed according to the length of a time elapsed since the display ofthe information display object.

For example, referring to FIG. 14, information display objects DSH1 andDSH2 are subjected to the change process of the display mode at timingsafter long elapsed times TE1 and TE2. On the other hand, an informationdisplay object DSH3 is subjected to the change process of the displaymode at a timing after a short elapsed time TE3. In this case, theinformation display objects DSH1 and DSH2 with the long elapsed timesTE1 and TE2 since the display are subjected to the change process of thedisplay mode such as erase or semi-transparentization. On the otherhand, the information display object DSH3 with the short elapsed timeTE3 since the display is not subjected to the change process of thedisplay mode such as erase or semi-transparentization.

For example, as for the information display objects DSH1 and DSH2 withthe long elapsed times TE1 and TE2 since the display, there is a highpossibility that the user has already seen the information displayobjects DSH1 and DSH2 during the long elapsed times TE1 and TE2, and itis thus considered that information transmission from the informationdisplay objects DSH1 and DSH2 has been sufficiently performed.Therefore, the information display objects DSH1 and DSH2 are subjectedto the change process of the display mode such as erase orsemi-transparentization when the change condition is satisfied.

On the other hand, as for the information display object DSH3 with theshort elapsed time TE3 since the display, there is a low possibilitythat the user has sufficiently seen the information display object DSH3during the short elapsed time TE3, and it is thus considered thatinformation transmission from the information display object DSH3 isinsufficient. Therefore, the information display object DSH3 is notsubjected to the change process of the display mode such as erase orsemi-transparentization even when the change condition is satisfied.

Specifically, while a large number of information display objects aredisplayed, if all the information display objects are erased orsemi-transparentized when the line-of-sight direction of the userchanges greatly and the change condition is satisfied, informationtransmission from the information display objects will be insufficient.In this respect, according to the method illustrated in FIG. 14, it isdetermined whether to perform the change process of the display modebased on the elapsed time since the display, which makes it possible toprevent this problem in an effective manner.

3.3 Determination on Change Condition

Next, a specific example of determination on the change condition willbe described. In the present embodiment, the determination section 115illustrated in FIG. 1 determines whether the change condition issatisfied. When the determination section 115 determines that the changecondition is satisfied, the change process of the display mode of theinformation display object is performed. Specifically, the determinationsection 115 determines whether a given change condition is satisfiedbased on the point-of-view change information of the user. Thepoint-of-view change information is line-of-sight change information orpoint-of-view position change information, for example. Specifically,when the change angle of the line-of-sight direction of the user becomesequal to or greater than a given angle, when the change speed of theline-of-sight direction becomes equal to or higher than a given speed,or when the change acceleration of the line-of-sight direction becomesequal to or higher than a given acceleration, the determination section115 determines that the given change condition is satisfied.

For example, referring to FIG. 15A, the line-of-sight direction VL ofthe user PL changes upward. In addition, a change angle Δθ of theline-of-sight direction VL is equal to or greater than an angle θth as athreshold (equal to or greater than a given angle). In this case, it isdetermined that the change condition is satisfied, and the changeprocess of the display mode of the information display object isperformed.

Referring to FIG. 15B, a change speed ω of the line-of-sight directionof the user PL is equal to or higher than a speed ωth as a threshold(equal to or higher than a given speed). Otherwise, an acceleration α ofthe line-of-sight direction of the user PL is equal to or higher than anacceleration αth as a threshold (equal to or higher than a givenacceleration). In this case, it is determined that the change conditionis satisfied, and the change process of the display mode of theinformation display object is performed. The change speed ω and thechange acceleration α are respectively equivalent to an angular speedand an angular acceleration.

According to the method illustrated in FIG. 15A and FIG. 15B, when theline-of-sight direction VL of the user PL changes greatly or abruptly,it is determined that the change condition is satisfied and the changeprocess of the display mode of the information display object isperformed, thereby enabling the appropriate display process of theinformation display object.

In the present embodiment, as illustrated in FIG. 16, the content of thechange process of the display mode may be differentiated according tothe change angle Δθ, the change speed ω, or the change acceleration α ofthe line-of-sight direction of the user (point-of-view changeinformation in a broad sense).

For example, in the erase process of the information display object, thetime before the erasing of the information display object isdifferentiated according to the change angle Δθ, the change speed ω, orthe change acceleration α. For example, when the change angle Δθ, thechange speed ω, or the change acceleration α is large, the time beforethe erasing of the information display object is shortened so that theinformation display object is erased immediately. On the other hand,when the change angle Δθ, the change speed ω, or the change accelerationα is small, the time before the erasing of the information displayobject is lengthened.

In the semi-transparentization process of the information displayobject, the change speed (change rate) of semi-transparency isdifferentiated according to the change angle Δθ, the change speed ω, orthe change acceleration α. For example, when the change angle Δθ, thechange speed ω, or the change acceleration α is large, the change speedof semi-transparency is accelerated so that the information displayobject immediately comes close to transparency. On the other hand, whenthe change angle Δθ, the change speed ω, or the change acceleration α issmall, the change speed of semi-transparency is decelerated so that theinformation display object gradually comes close to transparency.

Accordingly, the content of the change process of the display mode ischanged according to the change angle, the change speed, or the changeacceleration of the line-of-sight direction, thereby implementing thechange process of the display mode reflecting the change angle, thechange speed, or the change acceleration.

In the change process of the display mode by the movement processillustrated in FIG. 11A and FIG. 11B, for example, the movement speed ofthe information display object is differentiated according to the changeangle Δθ, the change speed ω, or the change acceleration α. In thechange process of the display mode by the size change processillustrated in FIG. 12A and FIG. 12B, the size change speed of theinformation display object is differentiated according to the changeangle Δθ, the change speed ω, or the change acceleration α.

The magnitudes of the change angle Δθ, the change speed ω, and thechange acceleration α can be the magnitudes of differences from thethresholds θth, ωth, and αth, for example. The magnitudes of thedifferences are the magnitudes of Δθ−0th, ω−ωth, and α−αth, for example.For example, the change process of the display mode such as the eraseprocess or semi-transparentization process of the information displayobject is performed on the premise that Δθ−θth, ω−ωth, and α−αth are 0or greater.

FIG. 17A and FIG. 17B are diagrams describing a method for determining achange condition different from that described in FIGS. 15A and 15B.According to this determination method, as illustrated in FIG. 17A, adetermination range RG (hit range or hit box) is set for the informationdisplay object DS. Specifically, the determination range RG is set toinclude the information display object DS as seen from the point-of-viewof the user. As illustrated in FIG. 17B, when a gaze position GP in theline-of-sight direction VL of the user PL exceeds the determinationrange RG (acceptance range in the line-of-sight direction), it isdetermined that the change condition is satisfied.

Specifically, crossing determination is performed to determine whether astraight line along the line-of-sight direction VL (straight lineconnecting VP and GP) and the determination range RG cross each other.The crossing determination can be implemented by a publicly knowncollision process (hit determination). When the straight line along theline-of-sight direction VL crosses the determination range RG, no changeprocess of the display mode of the information display object DS isperformed. On the other hand, as illustrated in FIG. 17B, when thestraight line along the line-of-sight direction VL does not cross thedetermination range RG, the change process of the display mode of theinformation display object DS is performed. That is, when the gazeposition GP at an end of the straight line is beyond the determinationrange RG (outside the determination range RG), the change process of thedisplay mode of the information display object DS is performed.

Accordingly, it is possible to use the information display object DS asa reference to detect a relative change in the line-of-sight directionVL to the information display object DS, thereby determining the changecondition for the change process of the display mode.

In the case of setting the determination range for the informationdisplay object, the determination range may be differentiated for eachof the information display objects. For example, referring to FIG. 18, adetermination range RG1 (first determination range) is set for aninformation display object DS1 (first information display object). Adetermination range RG2 (second determination range) different in sizefrom the determination range RG1 is set for an information displayobject DS2 (second information display object). A determination rangeRG3 (third determination range) different in size from the determinationranges RG1 and RG2 is set for an information display object DS3 (thirdinformation display object).

Accordingly, it is possible to set different determination rangesaccording to information display objects to determine the changecondition based on different determination criteria. For example,referring to FIG. 18, the smaller determination range RG1 is set for theinformation display object DS1 positioned in the central part as seenfrom the point-of-view of the user. Therefore, as obvious from FIG. 17B,the change process of the display mode of the information display objectDS1 is performed even with a small change in the line-of-sight directionVL (a change relative to the determination range). On the other hand,the larger determination ranges RG2 and RG3 are set for the informationdisplay objects DS2 and DS3 positioned at a peripheral part (peripheraledge part) as seen from the point-of-view of the user. Therefore, nochange process of the display mode of the information display objectsDS2 and DS3 is performed with a small change in the line-of-sightdirection VL, and when the line-of-sight direction VL changes greatly,the change process of the display mode is performed. In contrast to thecase illustrated in FIG. 18, a larger determination range may be set forthe information display object DS1 positioned in the central part, andsmaller determination ranges may be set for the information displayobjects DS3 and DS4 positioned at the peripheral part.

In the present embodiment, when a plurality of information displayobjects are arranged in the virtual space, the change process of thedisplay mode may be performed on the information display objectpositioned in the central part as seen from the point-of-view of theuser. For example, referring to FIG. 19, the information display objectDS1 is positioned in the central part (center) as seen from thepoint-of-view of the user. The central part is a range of apredetermined size centered on the gaze position of the user in theline-of-sight direction, for example. On the other hand, the informationdisplay objects DS2, DS3, DS4, and DS5 are positioned at the peripheralpart (peripheral edge part) as seen from the point-of-view of the user.The peripheral part is a region around the central part, and the centralpart and the peripheral part are separated by a predetermined boundary.

When the plurality of information display objects DS1 to DS5 arearranged as illustrated in FIG. 19, the change process of the displaymode (erasing or semi-transparentization) is performed on theinformation display object DS1 in the central part. On the other hand,no change process of the display mode is performed on the informationdisplay objects DS2 to DS5 at the peripheral part. For example, the usermoves the line-of-sight direction to look around the vast VR space. Inthis situation, if the large-sized information display object DS1remains on normal display in the central part as illustrated in FIG. 19,the information display object DS1 blocks the field of view of the user.

In this respect, in the example of FIG. 19, the change process of thedisplay mode is performed on the information display object DS1 at thecentral part on a priority basis. Accordingly, the information displayobject DS1 is erased or semi-transparentized in the field of view of theuser looking round the VR space, which effectively prevents theinformation display object DS1 from blocking the field of view of theuser.

In the present embodiment, when a plurality of information displayobjects are arranged in a virtual space, the content of the changecondition and the content of the change process of the display mode maybe differentiated for each of the information display objects. Forexample, the content of the change condition or the content of thechange process of the display mode is differentiated between a firstinformation display object and a second information display object outof the plurality of information display objects.

For example, referring to FIG. 20, the plurality of information displayobjects DS1, DS2, and DS3 are arranged. For the information displayobject DS1, the change condition is set to CD1 and the change process ofthe display mode is set to PR1. Similarly, for the information displayobjects DS2 and DS3, the change conditions are respectively set to CD2and CD3, and the change processes of the display mode are respectivelyset to PR2 and PR3. The contents of the change conditions CD1, CD2, andCD3 are different from one another. The contents of the change processesof the display mode PR1, PR2, and PR3 are different from one another.

For example, as for the change conditions CD1, CD2, and CD3, thethresholds θth, ωth, and αth of the change angle Δθ, the change speed ω,and the change acceleration α in the line-of-sight direction VLillustrated in FIG. 15A and FIG. 15B are different from one another.Otherwise, as for the change conditions CD1, CD2, and CD3, asillustrated in FIG. 18, the sizes of the determination ranges set forthe information display objects are different from one another. As forthe change processes of the display mode PR1, PR2, and PR3, the lengthof a time before erasing, the change speed of semi-transparency, themovement speed, and the size change speed are different from oneanother, for example.

Accordingly, it is possible to set the appropriate contents of thechange conditions and the change processes according to the types,priorities, importance levels or roles of the plurality of informationdisplay objects to execute the change process of the display mode.

In the present embodiment, the change process of the display mode may beperformed according to a depth value of the information display object.For example, referring to FIG. 21A, depth values (Z values) of theinformation display objects DS1 and DS2 from the point-of-view VP of theuser PL are respectively Z1 and Z2. In this case, Z1<Z2 and theinformation display object DS1 is positioned closer to the front sidethan the information display object DS2 as seen from the point-of-viewVP of the user PL.

Then, the line-of-sight direction VL of the user PL changes greatly andthe change condition is satisfied as illustrated in FIG. 21B. In thiscase, referring to FIG. 21B, the change process of the display mode isperformed on the information display object DS1 of which the depth valueZ1 is small. On the other hand, no change process of the display mode isperformed on the information display object DS2 of which the depth valueZ2 is large. That is, it is selected whether the change process of thedisplay mode is to be performed or not to be performed when the changecondition is satisfied according to the magnitude of the depth value.

For example, when the user PL changes greatly the line-of-sightdirection VL to look around the VR space, the information display objectDS1 of which the depth value Z1 is small and that is positioned closerto the front side blocks the field of view of the user. On the otherhand, the information display object DS2 of which the depth value Z2 islarge and that is positioned closer to the back side is considered asbeing less prone to block the field of view of the user. Accordingly,referring to FIG. 21B, the information display object DS1 of which thedepth value Z1 is small is subjected to the change process of thedisplay mode to erase or semi-transparentize the information displayobject DS1. On the other hand, the information display object DS1 ofwhich the depth value Z2 is large is not subjected to the change processof the display mode but is normally displayed. This makes it possible toimplement the change process of the display mode of the informationdisplay object effectively using the depth value of the informationdisplay object.

In the present embodiment, the change process of the display mode may beperformed according to the length of a time elapsed since the changecondition is satisfied.

For example, FIG. 22A, FIG. 22B, and FIG. 22C illustrate the states inwhich the times elapsed since the change condition is satisfied arerespectively TF1, TF2, and TF3. In this case, TF3>TF2>TF1, and FIG. 22Aillustrates the case in which the elapsed time TF1 is short, and FIG.22C illustrates the case in which the elapsed time TF3 is long.

Referring to FIG. 22A, the user PL changes greatly the line-of-sightdirection VL upward, and as the change process of the display mode, thesemi-transparentization process of the information display object DS1 isperformed. In this case, the degree of semi-transparency is AU.Referring to FIG. 22B, the user PL changes greatly the line-of-sightdirection VL downward, and the semi-transparentization process of theinformation display object DS2 is performed. In this case, the degree ofsemi-transparency is AL2. Referring to FIG. 22C, the user PL changesgreatly the line-of-sight direction VL upward, and thesemi-transparentization process of the information display object DS1 isperformed. In this case, the degree of semi-transparency is AL3.

In this manner, referring to FIG. 22A to FIG. 22C, the user PL changesgreatly the line-of-sight direction VL upward or downward to look aroundthe VR space. At this time, the change in the line-of-sight direction VLis large and thus time elapses while the change condition remainssatisfied. In this case, referring to FIG. 22A to FIG. 22C, the degreesof semi-transparency AL1, AL2, and AL3 are set such that the informationdisplay objects DS1 and DS2 gradually come closer to transparencyaccording to the length of the time elapsed. For example, the degree ofsemi-transparency AL3 with the elapsed time TF3 illustrated in FIG. 22Cis set to be closer to transparency than the degree of semi-transparencyAL1 with the elapsed time TF1 illustrated in FIG. 22A.

Accordingly, when the user looks around the VR space and the changecondition is satisfied, the information display object gradually comescloser to transparency with the lapse of time. Then, after a sufficientlength of time has elapsed, the information display object is almosttransparent so that the user can look around the VR space without thefield of view being blocked by the information display object.

3.4 Return Process

Next, a return process of the change process of the display mode will bedescribed. In the present embodiment, after the change process of thedisplay mode of the information display object is performed, when it isdetermined that a return condition is satisfied, a return process of thechange process of the display mode is performed.

For example, referring to FIG. 23A, the information display object DS isarranged in the line-of-sight direction VL of the user PL. Referring toFIG. 23B, the line-of-sight direction VL of the user PL changes greatlyand the change condition is satisfied, and thus the change process ofthe display mode of the information display object DS is performed.Specifically, the erase process or semi-transparentization process ofthe information display object DS is performed.

Referring to FIG. 23C, after the change process of the display mode isperformed, the return condition is satisfied due to the lapse of a giventime, for example, and the return process of the change process of thedisplay mode is performed. Specifically, the return process is performedsuch that the information display object DS erased as illustrated inFIG. 23B is displayed again. Otherwise, the return process is performedsuch that the semi-transparentization process of the information displayobject DS illustrated in FIG. 23B is canceled and the informationdisplay object DS is normally displayed.

Referring to FIG. 24A and FIG. 24B, the line-of-sight direction VL ofthe user PL changes greatly and the change condition is satisfied, andthus as the change process of the display mode of the informationdisplay object DS, the cancel process of following the line-of-sightdirection VL is performed. That is, the change process of the displaymode described above with reference to FIG. 10A and FIG. 10B isperformed.

Referring to FIG. 24C, after the change process of the display mode isperformed, the return condition is satisfied due to the lapse of a giventime, for example, and thus the return process of the change process ofthe display mode is performed. Specifically, the cancel process offollowing illustrated in FIG. 24B is disabled to cause the informationdisplay object DS to follow the line-of-sight direction VL.

Performing the return process makes it possible to, when the changecondition is satisfied and the change process of the display mode of theinformation display object is performed, prevent the display mode of theinformation display object from remaining in the changed state. Then,the information display object can be returned to the normal displaystate before the change process of the display mode.

FIG. 25 is a flowchart illustrating a detailed example of the returnprocess. First, it is determined whether a given change condition issatisfied (step S11), and when the change condition is satisfied, thechange process of the display mode of the information display object isperformed (step S12). Specifically, the erase process, thesemi-transparentization process, or the cancel process of following asillustrated in FIG. 23B and FIG. 24B is performed.

Then, it is determined whether a given time has elapsed after the changeprocess of the display mode, or whether no change in the line-of-sightdirection of the user has been detected for a given time after thechange process of the display mode, or whether gaze of the user on anobject has been detected after the change process of the display mode(step S13). That is, it is determined whether the return condition issatisfied. It is determined that the return condition is satisfied whena given time has elapsed, or when no change in the line-of-sightdirection has been detected for a given time, or when the gaze of theuser on an object has been detected. Then, the return process of thechange process of the display mode is performed (step S14). That is, thereturn process as illustrated in FIG. 23C or FIG. 24C is performed.

For example, when a given time has elapsed after the change process ofthe display mode, there is no problem in returning to the normaldisplay, and thus the return process of the change process of thedisplay mode is performed. For example, when the user looks around theVR space, returning the information display object to the normal displayafter the lapse of a sufficient length of time is not considered toimpede the user's looking around the VR space.

When no change in the line-of-sight direction of the user has beendetected for a given time after the change process of the display mode,it is considered that the user has kept watching a desired place in theVR space for a given period of time without moving the line-of-sightdirection. Therefore, in this case, it is considered that the user hassufficiently watched the desired place, and thus the return process ofthe change process of the display mode is performed. For example, theerased information display object is displayed again or thesemi-transparent information display object is returned to normaldisplay.

When the gaze of the user on an object has been detected after thechange process of the display mode, it is considered that the user hasgazed at the desired object (existing around the user) in the VR space.Therefore, in this case, it is considered that the user has sufficientlywatched the desired object, and thus the return process of the changeprocess of the display mode is performed. In this manner, consideringvarious return conditions makes it possible to determine the appropriatereturn condition corresponding to the user's various situations toexecute the return process.

In the present embodiment, the brightness of the display image on theHMD may be differentiated between when the change process of the displaymode of the information display object is not performed and when thechange process of the display mode of the information display object isperformed. For example, the brightness of the display image is changedbetween when the information display object is normally displayed andwhen the information display object is not normally displayed.

For example, FIG. 26A illustrates a state in which no change process ofthe display mode of the information display object is performed. In thiscase, the information display object is not subjected to the eraseprocess or semi-transparentization process but is on normal display. Atthis time, a display image IMG on the HMD is darkened, for example. Thatis, the brightness of the entire display image IMG is lowered.

On the other hand, FIG. 26B illustrates a state in which the changeprocess of the display mode of the information display object isperformed. Referring to FIG. 26B, the user PL changes the line-of-sightdirection VL upward and downward to look around the VR space. When theline-of-sight direction VL changes in this manner, the change conditionis satisfied and the change process of the display mode such as eraseprocess or semi-transparentization process is performed on theinformation display object. At this time, the display image IMG on theHMD is brightened, for example. That is, the brightness (luminance) ofthe entire display image IMG is higher than that in FIG. 26A.

When the user PL turns the line-of-sight direction VL to the frontdirection as illustrated in FIG. 26A, for example, it is appropriate todisplay a large number of information display objects as illustrated inFIG. 4 to FIG. 6 to transmit various kinds of information to the userPL. Accordingly, referring to FIG. 26A, no change process of the displaymode of the information display object is performed but the informationdisplay object is on normal display.

On the other hand, as illustrated in FIG. 26B, when the user PL changesthe line-of-sight direction VL upward, downward, leftward, and rightwardto look around the VR space, it is appropriate to ensure the field ofview of the user PL. Accordingly, referring to FIG. 26B, the eraseprocess or semi-transparentization process of the information displayobject is performed so that the information display object does notblock the field of view of the user PL looking around the VR space.

In addition, referring to FIG. 26B, the entire display image IMG isbrightened. Accordingly, it is possible to allow the user PL to lookaround the VR space with the bright display image IMG, thereby enablingthe generation of the more appropriate display image on the HMD.

3.5 Command Process

Next, detailed descriptions of the command process in the presentembodiment described above with reference to FIG. 6 will be provided. Inconventional communication games using characters and the like,two-dimensional games are mainstream and no consideration is given tothe positional relationship and the line-of-sight relationship betweenthe user and the character. Therefore, when the user uses the commandCM3 “speak” illustrated in FIG. 6, for example, the effect activated bythe command CM3 is the same regardless whether the distance between theuser and the character is short or long. Therefore, the user cannot feela sense of distance to the character, which causes a problem that it isnot possible to improve the virtual reality for the user.

Accordingly, in the present embodiment, the effects of commands arechanged depending on the positional relationship, the line-of-sightrelationship and the gaze situation between the user and the character.In the present embodiment, for example, the effects of commands areactivated according to the information on the character as a target ofgameplay by the user and the point-of-view information of the user. Forexample, the degrees of effects of commands and the contents of theeffects are changed according to the information of the character andthe point-of-view information of the user. Specifically, the effects ofcommands are activated according to the information of the charactersuch as position information, direction information, game parameterinformation, body part information, posture information, or typeinformation of the character, and the point-of-view information such aspoint-of-view position information, line-of-sight direction informationof the user, or virtual camera attribute information.

FIG. 27A to FIG. 28B are diagrams describing methods for activating theeffects of commands according to the positional relationship informationbetween the user PL and the character CH.

For example, referring to FIG. 27A and FIG. 27B, the degree of theeffect of a command is changed according to the positional relationshipinformation between the user PL and the character CH. In this case, thepositional relationship information indicates a distance LC between theuser PL and the character CH, and the distance LC is determined based ona point-of-view VP as the point-of-view information of the user PL and aposition PC as information of the character CH. The positionalrelationship information may be information indicating the relationshipbetween the direction of the user PL and the direction of the characterCH.

Referring to FIG. 27A and FIG. 27B, the user selects and uses thecommand CM1 “concentrate more!”, but the distance LC between the user PLand the character CH is long in FIG. 27A. On the other hand, thedistance LC is short in FIG. 27B.

In this case, in the present embodiment, even when the user uses thesame command CM1 as illustrated in FIG. 27A and FIG. 27B, the degree ofthe effect activated by the command CM1 is changed according to thedistance LC. For example, the distance LC is long in FIG. 27A and thedegree of the effect of the command CM1 becomes lower accordingly, andthe distance LC is short in FIG. 27B and the degree of the effect of thecommand CM1 becomes higher.

Specifically, when the command CM1 “concentrate more!” is used, thedistance LC is long in FIG. 27A and the degree of increasing the gameparameter for the concentration of the character CH is low as indicatedwith A1. In contrast to this, the distance LC is short in FIG. 27B andthe degree of increasing the game parameter for the concentration of thecharacter CH is higher as indicated with A2.

Accordingly, in the case of using the command CM1 “concentrate more!”,the user can approach the character CH and use the command CM1 tofurther enhance the concentration of the character CH. Therefore, it ispossible to allow the user to feel as if he or she got closer to thecharacter CH and said “concentrate more!”, thereby improving the virtualreality for the user.

Referring to FIG. 28A and FIG. 28B, the content of the effect of acommand is changed according to the positional relationship informationbetween the user PL and the character CH. That is, the degree of theeffect is changed in the example of FIG. 27A and FIG. 27B, whereas thecontent of the effect is changed in the example of FIG. 28A and FIG.28B.

For example, referring to FIG. 28A, the user uses the command CM3“speak” when the distance LC to the character CH is long. In this case,as an utterance of the user to “speak”, for example, the word “Yoo-hoo!”is output by sound or displayed in a telop (balloon) on the game screen.

On the other hand, referring to FIG. 28B, the user uses the command CM3“speak” when the distance LC to the character CH is short. In this case,as an utterance of the user to “speak”, for example, the words “Are youOK?” or “Are you working hard?” are output by sound or is displayed in atelop on the game screen.

That is, in the case of FIG. 28A, the content of the effect of thecommand CM3 “speak” is the output of the word “Yoo-hoo!” by sound(displayed in a telop). On the other hand, in the example of FIG. 28B,with a decrease in the distance LC, the content of the effect is changedto the output of the words “Are you OK?” and “Are you working hard?” bysound.

According to this, even with the same command CM3 “speak”, the contentof the effect of the command is changed according to the positionalrelationship between the user PL and the character CH. Therefore, it ispossible to improve the virtual reality for the user and the degree ofvariety of game representations.

FIG. 29A and FIG. 29B are diagrams describing a method for activatingthe effect of a command according to line-of-sight relationshipinformation between the user PL and the character CH.

For example, referring to FIG. 29A and FIG. 29B, the degree and contentof the effect of a command are changed according to the line-of-sightrelationship information between the user PL and the character CH.Referring to FIG. 29A and FIG. 29B, the line-of-sight relationshipinformation is information representing the relationship among theline-of-sight direction VL of the user PL, and the position PC and theline-of-sight direction VLC of the character CH.

In addition, referring to FIG. 29A and FIG. 29B, the user selects anduses the command CM5 “offer advice”, but the line-of-sight direction VLof the user PL is not oriented to the position PC of the character CH inFIG. 29A. In contrast to this, in FIG. 29B, the line-of-sight directionVL of the user PL is oriented to the position PC of the character CH. Inaddition, in FIG. 29A, the line-of-sight direction VL of the user PL isnot aligned with the line-of-sight direction VLC of the character CH. Incontrast to this, in FIG. 29B, the line-of-sight directions VL and VLCare aligned with each other.

In this case, in the present embodiment, even when the user uses thesame command CM5, the degree and content of the effect activated by thecommand CM2 are changed according to the line-of-sight relationshipinformation as illustrated in FIG. 29A and FIG. 29B.

For example, in the case of the line-of-sight relationship illustratedin FIG. 29A, as the content of the effect activated by the command CM5“offer advance”, “normal advice” is offered. For example, ordinaryadvice is offered in a normal tone. As the effect of “normal advice” bythe command CM5, the motivation as a game parameter of the character CHincreases but the degree of the increase is low as indicated with B1 inFIG. 29A.

On the other hand, in the case of the line-of-sight relationshipillustrated in FIG. 29B, as the content of the effect of the command CM5“offer advice”, “tender advice” is offered. For example, encouragingadvice is offered in a tender tone. As the effect of “tender advice” bythe command CM5, the motivation of the character CH increases and thedegree of the increase is high as indicated with B2 in FIG. 29B.

Accordingly, by using the command CM5 “offer advice” while properlylooking at the character CH as illustrated in FIG. 29B, the user canoffer better advice and increase the degree of the increase of themotivation as the effect of the advice, as compared to the case in whichthe user uses the command CM5 without looking at the character CH asillustrated in FIG. 29A. Therefore, it is possible to allow the user tofeel as if he or she offers advice to the opposite-sex person in frontof him or her, thereby significantly improving the virtual reality forthe user.

FIG. 30 is a diagram describing a method for activating the effect of acommand according to gaze information of the user. For example,referring to FIG. 30, the effect of the command is activated accordingto gaze part information as information on the body part of thecharacter CH gazed at by the user.

Specifically, referring to FIG. 30, the user uses a command CM4 “poke”.In this case, as indicated with C1 in FIG. 30, the user uses the commandCM4 “poke” while gazing at the shoulder of the character CH as a bodypart. In this case, an image (video image) representing a scene in whichthe user pokes the shoulder of the character CH is displayed. Forexample, an image representing that the user's hand (object) appears onthe screen and the hand pokes the shoulder of the character CH isdisplayed. On the other hand, as indicated with C2 in FIG. 30, the useruses the command CH4 “poke” while gazing at the head (forehead) of thecharacter CH as a body part. In this case, an image representing a scenein which the user pokes the head of the character CH is displayed. Forexample, an image representing that the user's hand appears on thescreen and the hand pokes the head of the character CH is displayed.

Accordingly, even when the same command CM4 “poke” is used, the action“poke” is performed in different ways according to the body part of thecharacter CH gazed at by the user. In addition, the user's action isperformed on the body part gazed at by the user, which allows the userto feel as if he or she actually pokes the character CH, therebyimproving the virtual reality for the user.

According to the methods for the command process in the presentembodiment as described above, the virtual reality for the user can befurther improved. On the other hand, according to the methods for thecommand process in the present embodiment, a large number of informationdisplay objects are arranged and displayed as illustrated in FIG. 6 andFIG. 30. Therefore, when the user looks around the VR space, if thelarge number of information display objects are normally displayed,these information display objects block the field of view of the user.In this respect, in the present embodiment, when the change condition issatisfied by a change in the line-of-sight direction, the process ofchanging the display mode of the information display object isperformed, thereby enabling display of the more appropriate displayimage on the HMD.

3.6 Process Example

Next, an example of the process according to the present embodiment isdescribed with reference to a flowchart in FIG. 31. First, trackinginformation of the point-of-view information of the user is acquired(step S21). For example, the tracking information obtained through thetracking process described with reference to FIG. 2A to FIG. 3B isacquired. Then, based on the acquired tracking information, the virtualcamera is controlled to follow a change in the point-of-view of the user(step S22). For example, the virtual camera is controlled such that,when the user turns the line-of-sight upward, downward, leftward, orrightward in the real world, the line-of-sight direction of the virtualcamera also turns upward, downward, leftward, or rightward accordinglyin the virtual space. In addition, the virtual camera is controlled suchthat, when the point-of-view position of the user moves upward,downward, leftward, or rightward in the real world, the point-of-viewposition of the virtual camera also moves upward, downward, leftward, orrightward accordingly in the virtual space.

Next, it is determined whether a given change condition is satisfied bya change in the line-of-sight direction of the user (step S23). Forexample, it is determined whether the change condition is satisfied bythe methods described above with reference to FIG. 15A, FIG. 15B, FIG.17A, and FIG. 17B. When the change condition is satisfied, the changeprocess of the display mode of the information display object isperformed (step S24). For example, the change process of the displaymode described above with reference to FIG. 9A to FIG. 12B is performed.Then, an image as viewed from the virtual camera in the virtual space isgenerated and displayed on the HMD (step S25). For example, as images asviewed from the virtual camera, a left-eye image and a right-eye imagefor stereoscopic viewing are generated and displayed on the HMD.

Although the present embodiments have been described in detail above,those skilled in the art will readily appreciate that many modificationsare possible in the embodiment without materially departing from thenovel teachings and advantages of the present disclosure. Accordingly,all such modifications are intended to be included within the scope ofthe present disclosure. For example, each of terms (such as a change inthe line-of-sight direction, a character, or a book) that is at leastonce used together with another broader-sense or synonymous term (suchas a change in point-of-view, a moving body, or an item) in thespecification or the drawings can be replaced with the other term at anypart of the specification or the drawings. In addition, the settingprocess of a virtual space, the control process of a virtual camera, thearrangement process of an information display object, the change processof a display mode, the determination process of a change condition, andthe determination process of a return condition are not limited to thosedescribed above in relation to the present embodiments, and methods,processes, and configurations equivalent to them are included in therange of the present disclosure. The embodiments can be applied tovarious games. The embodiments can be applied to various simulationsystems such as an arcade game apparatus, a consumer game apparatus, alarge attraction system where a large number of users play the game, andthe like.

What is claimed is:
 1. A simulation system comprising a processorincluding hardware, the processor being configured to perform: a virtualspace setting process of setting a virtual space in which a plurality ofobjects are arranged, the plurality of objects including a plurality ofhead-locked information display objects and other objects, the pluralityof head-locked information display objects and the other objects havingthree-dimensional coordinate values indicating three dimensionalpositions in the virtual space, the three-dimensional coordinate valuesof the plurality of head-locked information display objects and theother objects being stored in a memory as an object information; avirtual camera control process of controlling a virtual cameracorresponding to a point-of-view of a user wearing a head mounteddisplay; a display process of generating an image as viewed from thevirtual camera in the virtual space as a display image on the headmounted display; and a determination process of determining whether agiven change condition is satisfied based on point-of-view changeinformation of the user, wherein: in the virtual space setting process,the processor performs arranging the plurality of head-lockedinformation display objects, which are controlled to follow a change inthe point-of-view of the user in the virtual space, each of theplurality of head-locked information display objects being arranged at agiven depth position in a line-of-sight direction of the user in thevirtual space, which is a different depth position from the otherobjects in the virtual space, the depth position of each of theplurality of head-locked information display objects being a coordinatevalue of a Z-coordinate axis in a camera coordinate system of thevirtual camera, when the line-of-sight direction of the user changes,the three-dimensional position of each of the plurality of head-lockedinformation display objects in the virtual space changes accordinglywhile the three dimensional positions of the other objects in thevirtual space remain unchanged, in the determination process, theprocessor performs determining that the given change condition issatisfied when a change speed of the line-of-sight direction or apoint-of-view position of the user is equal to or higher than a givenspeed, or when a change in acceleration of the line-of-sight directionor the point-of-view position of the user is equal to or higher than agiven acceleration, and in the display process, (i) the processorperforms an erase process of at least one of the plurality ofhead-locked information display objects, a semi-transparentizationprocess of at least one of the plurality of head-locked informationdisplay objects, or a cancel process of following by at least one of theplurality of head-locked information display objects to a change in thepoint-of-view of the user, as a change process of a display mode of theat least one of the plurality of head-locked information displayobjects, when determining that the given change condition is satisfied,and (ii) the processor performs a return process of the change processof the display mode of the plurality of head-locked information displayobjects after the change process of the display mode of the at least oneof the plurality of head-locked information display objects, upondetermining in the determination process that a given return conditionis satisfied, and while the at least one head-locked information displayobject of the plurality of head-locked information display objects issubjected to the change process of the display mode and the returnprocess, a remaining of the plurality of head-locked information displayobjects and the other objects are not subjected to the change process ofthe display mode and the return process.
 2. The simulation system asdefined in claim 1, wherein in the display process, as the change speedor the change acceleration of the line-of-sight direction or thepoint-of-view position is higher, the processor performs shortening atime before the at least one head-locked information display object iserased, or accelerating a change speed of semi-transparency in thesemitransparentization process, or shortening a time before thefollowing by the at least one head-locked information display object iscanceled.
 3. The simulation system as defined in claim 1, wherein in thedetermination process, the processor performs determining that the givenreturn condition is satisfied when a given time has elapsed after thechange process of the display mode, or when no change in theline-of-sight direction of the user has been detected during the giventime after the change process of the display mode, or when a gaze of theuser on a given object has been detected after the change process of thedisplay mode.
 4. The simulation system as defined in claim 1, wherein inthe display process, the processor performs the change process of thedisplay mode according to type, priority, or importance level of the atleast one head-locked information display object.
 5. The simulationsystem as defined in claim 4, wherein in the display process, when theat least one head-locked information display object is an informationdisplay object for command selection, the processor performs no changeprocess of the display mode of the at least one head-locked informationdisplay object, and when the at least one head-locked informationdisplay object is an information display object for description display,the processor performs the change process of the display mode of the atleast one head-locked information display object.
 6. The simulationsystem as defined in claim 1, wherein in the display process, theprocessor performs the change process of the display mode according to alength of a time elapsed from the display of the at least onehead-locked information display object.
 7. The simulation system asdefined in claim 6, wherein in the display process, the processorperforms the change process of the display mode on a head-lockedinformation display object with a longer time elapsed from the displayin priority as compared to a head-locked information display object witha shorter time elapsed from the display.
 8. The simulation system asdefined in claim 1, wherein: in the display process, the processorperforms differentiating a content of the given change condition or acontent of the change process of the display mode between a firsthead-locked information display object and a second head-lockedinformation display object of the plurality of head-locked informationdisplay objects.
 9. The simulation system as defined in claim 8, whereinin the display process, the processor performs differentiating athreshold to determine the change condition based on the change speed orthe change acceleration of the line-of-sight direction or thepoint-of-view position between the first head-locked information displayobject and the second head-locked information display object.
 10. Thesimulation system as defined in claim 1, wherein in the display process,the processor performs the change process of the display mode accordingto a change in the depth position of the at least one head-lockedinformation display object.
 11. The simulation system as defined inclaim 10, wherein: a first head-locked information display object of theat least one head-locked information display object of the plurality ofhead-locked information display objects having a first depth positionand a second head-locked information display object of the plurality ofhead-locked information display objects having a second depth positionare arranged in the virtual space, and in the display process, theprocessor performs the change process of the display mode on the firsthead-locked information display object as a head-locked informationdisplay object on a front side as seen from the point-of-view, and theprocessor performs no change process of the display mode on the secondhead-locked information display object as a head-locked informationdisplay object on a back side as seen from the point-of-view.
 12. Thesimulation system as defined in claim 1, wherein in the display process,the processor performs the change process of the display mode accordingto a length of a time elapsed since the given change condition issatisfied.
 13. The simulation system as defined in claim 1, wherein inthe display process, the processor performs differentiating brightnessof the display image on the head mounted display between (i) when thechange process of the display mode of the at least one head-lockedinformation display object is not performed and (ii) when the changeprocess of the display mode of the at least one head-locked informationdisplay object is performed.
 14. The simulation system as defined inclaim 1, wherein at least one of the plurality of head-lockedinformation display objects is for description display, and at leastanother one of the plurality of head-locked information display objectsis for command selection.
 15. A processing method comprising: a virtualspace setting process of setting a virtual space in which a plurality ofobjects are arranged, the plurality of objects including a plurality ofhead-locked information display objects and other objects, the pluralityof head-locked information display objects and the other objects havingthree-dimensional coordinate values indicating three dimensionalpositions in the virtual space, the three-dimensional coordinate valuesof the plurality of head-locked information display objects and theother objects being stored in a memory as an object information; avirtual camera control process of controlling a virtual cameracorresponding to a point-of-view of a user wearing a head mounteddisplay; a display process of generating an image as viewed from thevirtual camera in the virtual space as a display image of the headmounted display; and a determination process of determining whether agiven change condition is satisfied based on point-of-view changeinformation of the user, wherein: in the virtual space setting process,arranging the plurality of head-locked information display objectscontrolled to follow a change in the point-of-view of the user in thevirtual space, each of the plurality of head-locked information displayobjects being arranged at a given depth position in a line-of-sightdirection of the user in the virtual space, which is a different depthposition from the other objects in the virtual space, the depth positionof each of the plurality of head-locked information display objectsbeing a coordinate value of a Z-coordinate axis in a camera coordinatesystem of the virtual camera, when the line-of-sight direction of theuser changes, the three-dimensional position of each of the plurality ofhead-locked information display objects in the virtual space changesaccordingly while the three dimensional positions of the other objectsin the virtual space remain unchanged, in the determination process,determining that the given change condition is satisfied when a changespeed of the line-of-sight direction or a point-of-view position of theuser is equal to or higher than a given speed, or when a change inacceleration of the line-of-sight direction or the point-of-viewposition of the user is equal to or higher than a given acceleration,and in the display process, (i) performing an erase process of at leastone of the plurality of head-locked information display objects, asemi-transparentization process of at least one of the plurality ofhead-locked information display objects, or a cancel process offollowing by at least one of the plurality of head-locked informationdisplay objects to a change in the point-of-view of the user, as achange process of a display mode of the at least one of the plurality ofhead-locked information display objects, upon determining that the givenchange condition is satisfied, and (ii) performing a return process ofthe change process of the display mode of the plurality of head-lockedinformation display objects after the change process of the display modeof the at least one of the plurality of head-locked information displayobjects, upon determining in the determination process that a givenreturn condition is satisfied, and while the at least one head-lockedinformation display object of the plurality of head-locked informationdisplay objects is subjected to the change process of the display modeand the return process, a remaining of the plurality of head-lockedinformation display objects and the other objects are not subjected tothe change process of the display mode and the return process.
 16. Anon-transitory computer-readable information storage medium storing aprogram causing a computer to perform the processing method as definedin claim 15.